Should AI Tell You How to Use Drugs? Lessons from Pharmacist Duty and Harm Reduction

Introduction      

At 12:21 a.m. on May 31, 2025, Sam Nelson asked ChatGPT whether Xanax could alleviate the nausea he felt after taking fifteen grams of kratom. ChatGPT warned the combination was dangerous––then recommended “0.25–0.5 mg Xanax only if symptoms feel intense” and offered to “help troubleshoot further” with suggestions about Benadryl combinations, timing, and food intake. Hours later, the 19-year-old University of California Merced psychology student died from central nervous system depression, his blood alcohol content at 0.125, with alprazolam and kratom in his system. A University of California San Francisco toxicologist reviewing the exchange said he would never recommend that someone using kratom take any dose of another central nervous system (CNS) depressant.

But Nelson did. Seeking to understand why, Nelson’s mother, attorney Leila Turner-Scott, reviewed forty hours of chat logs documenting an eighteen-month relationship in which her son came to view ChatGPT as “his best friend.” In May 2026, his parents sued OpenAI. But Nelson was far from the only person turning to a chatbot for health advice. A 2026 Kaiser Family Foundation tracking poll found that roughly one-third of U.S. adults had used artificial intelligence (AI) chatbots for physical-health information in the past year, with those aged 18 to 29 using AI for mental-health information at roughly three times the rate of adults aged 50 and over. The rules governing those responses are a life-or-death issue, and right now, no coherent framework supplies them. The FDA has declined to regulate consumer-facing health chatbots, and early litigation has only begun to test whether tort law can fill the gap.

This Essay argues that the legal foundation already exists––not in AI policy, but in harm reduction principles embedded in four decades of pharmacist duty doctrine. Harm reduction is a public health framework that accepts drug use as a reality and designs interventions to reduce associated harms rather than demanding abstinence as a precondition for engagement. A growing number of courts have, without ever naming the concept, built exactly this framework into pharmacist liability law: duties to warn patients of dangerous drug interactions based on information already possessed, to refuse clearly dangerous prescriptions while continuing to serve the patient, to provide complete and accurate information enabling safer use, and to recognize escalating risk patterns within ongoing relationships. These principles–refined through cases from Riff v. Morgan Pharmacy (Pa. Super. Ct. 1986) to Cottam v. CVS Pharmacy (Mass. 2002)–translate directly to AI systems now providing equivalent advice to millions of users daily. No scholar has mapped pharmacist duty doctrine’s harm reduction architecture onto AI liability. This Essay  does so, arguing that the resulting framework resolves the central tension in AI drug safety: how to protect users from dangerous advice without driving them toward worse alternatives.

The legal framework is taking shape not through comprehensive federal regulation, but through litigation and state enforcement. In May 2025, Garcia v. Character Technologies, Inc. (M.D. Fla. 2025) held that AI chatbot outputs may not qualify as protected speech and allowed strict product liability claims to proceed after a 14-year-old’s suicide. By December, forty-two state attorneys general had sent letters to thirteen AI companies documenting at least six fatalities and demanding safety reforms. Seven states enacted AI chatbot legislation in 2025.1

This crystallizing liability framework creates perverse incentives. Companies facing tort exposure but lacking clear guidance on “reasonable care” gravitate toward two responses: surveillance-heavy designs that log and monitor conversations for litigation defense, or blanket refusals that push users toward less reliable sources. Both responses fail. Surveillance undermines the privacy and anonymity that make AI appealing to people seeking drug information—often precisely those avoiding formal healthcare due to stigma or criminalization. Blanket refusals do not eliminate information seeking; they displace it to Reddit forums, TikTok videos, and friends with no pharmacological knowledge.

The fundamental reality is that people will ask. They have always asked––friends, pharmacists, strangers who seemed to know. AI has not created this behavior; the forty million daily health queries to ChatGPT alone show that it has merely become the latest destination. Any framework that begins from the premise that drug questions should be suppressed has already failed. 

What follows builds out that alternative in four parts. Part I traces the transformation of pharmacist duty from mechanical dispensing to clinical counseling, identifying the harm reduction principles courts have embedded without naming them. Part II maps the developing AI liability landscape. Part III develops the harm reduction framework, demonstrating how each principle corresponds to duties pharmacist law already recognizes. Part IV addresses implementation, including the harder question of how AI should handle information about illicit substances.

I.   The Pharmacist Duty 

Over the last fifty years, courts have gradually rejected the “warehouse” theory of pharmacy practice.2 Pharmacists are no longer mere dispensers of pills; they are clinical counselors who owe affirmative duties to the patients who rely on them. Those duties translate directly to AI systems now providing comparable advice.

The foundational case is Riff v. Morgan Pharmacy (Pa. Super. Ct. 1986). Patient Patricia Riff received a prescription for Cafergot, an ergotamine medication, with instructions to “insert one in the rectum ‘every four hours for headache,’” but no warning about maximum dosage. After taking dozens of the suppositories, Riff suffered toxic effects from overuse. The Superior Court of Pennsylvania held that each member of the healthcare team “has an affirmative duty to be, to a limited extent, his brother’s keeper.” Rejecting the defense that pharmacists are merely “shipping clerk[s] who must dutifully and unquestioningly obey the written orders of omniscient physicians,” the court found that when a prescription contains inadequate dosage instructions, the pharmacist must verify patient understanding or contact the prescriber. The jury apportioned 65% of the fault to the pharmacy––a striking allocation demonstrating that pharmacist liability can exceed prescriber liability when counseling failures occur. Riff’s core insight is that the pharmacist’s duty is relational and affirmative, arising not from a regulatory command but from the act of providing drug information to someone who will rely on it. An AI chatbot generating personalized drug advice is making precisely the same relational claim the Riff court recognized and, therefore, assumes the same responsibility.

Happel v. Wal-Mart Stores, Inc. (Ill. 2002) refined the pharmacist’s duty into a workable standard. Plaintiff Heidi Happel was allergic to aspirin and nonsteroidal anti-inflammatory drugs (NSAID)—a fact recorded in Wal-Mart’s computer system. When she presented a prescription for Toradol, an NSAID, the pharmacy’s computer generated a contraindication alert. The pharmacist overrode the alert without contacting the physician or warning Happel, who suffered an allergic reaction. The Illinois Supreme Court adopted a four-factor test weighing foreseeability, likelihood of injury, burden of the proposed duty, and consequences of imposing that burden. Finding the burden “minimal” because it only requires the pharmacist “telephone the physician and inform [them] of the contraindication . . . [or] provide the same information to the patient,” the court recognized a “narrow duty to warn” when the pharmacy possesses patient-specific contraindication information.

Happel’s framework has profound implications for AI liability. The duty arises from information the system already possesses, not from surveillance. Wal-Mart did not have to investigate Happel’s medical history; the allergy information was already in its database because she had previously provided it. Similarly, when a user tells ChatGPT he has taken fifteen grams of kratom and asks about adding Xanax, the system possesses information sufficient to trigger a duty––without logging conversations indefinitely or monitoring users across sessions. And the burden on an AI system is even more minimal than the burden the Happel court found acceptable. A pharmacist must pick up a telephone, while an AI chatbot need only generate a contraindication warning or display a crisis resource link. If the cost of a phone call satisfies Happel’s burden analysis, the cost of a programmatic safety response is negligible.

Hooks SuperX, Inc. v. McLaughlin (Ind. 1994) extended pharmacist duties to pattern recognition within existing relationships. The Indiana Supreme Court held that a pharmacist must cease refilling prescriptions for dangerous drugs when refill patterns occur “at an unreasonably faster rate than the rate prescribed.” The Hooks insight is temporal: pharmacists accumulate knowledge over repeated interactions with the same patient, and this accumulating knowledge triggers heightened duties. A pharmacist who fills a single problematic prescription may have no duty beyond the individual transaction; a pharmacist who fills the fifth early refill in a two-month period cannot claim ignorance of the pattern. AI chatbots with persistent memory or extended conversation histories accumulate analogous knowledge. Sam Nelson’s interactions with ChatGPT spanned eighteen months, during which the chatbot’s responses progressively degraded from refusals to explicit drug advice, culminating in a specific dose recommendation for a fatal combination. Hooks suggests that an AI system engaging in this kind of longitudinal relationship bears a duty to recognize escalating risk patterns within conversations, without requiring surveillance across all users or sessions.

Courts have also recognized that incomplete drug information can be worse than no information at all. In Cottam v. CVS Pharmacy (Mass. 2002), the Massachusetts Supreme Judicial Court held that a pharmacy voluntarily assuming a duty to provide drug information must provide complete and accurate information—resulting in a $357,000 verdict when CVS provided a “short form” warning for Trazodone that omitted a significant side effect. The Cottam principle has direct application to AI–generated drug advice. ChatGPT’s response to Sam Nelson illustrates Cottam’s concern precisely: the chatbot warned that combining kratom and Xanax was “dangerous,” then recommended a specific dose and offered to “troubleshoot” additional drug combinations. The warning created a false sense of managed risk, suggesting to Nelson that the danger had been identified and accounted for, when in fact no dose of a benzodiazepine like Xanax was safe given the substances already in his system. Under Cottam, this incomplete warning may generate greater liability than silence would have, because the partial information actively misled the user about the degree of risk.

Congress codified expanded pharmacist duties through the Omnibus Budget Reconciliation Act of 1990 (OBRA ’90). The statute responded to congressional findings that 30–50% of prescriptions failed to produce desired results because of improper use, and that annual drug-related morbidity and mortality costs exceeded $136 billion—much of it preventable through proper counseling. OBRA ’90’s implementing regulations mandate counseling on interactions and contraindications, establishing a duty that, in practice, requires the clinical judgment of a pharmacist. Most states extended these requirements beyond Medicaid to all patients, transforming counseling from discretionary professional judgment into legal obligation.

The learned intermediary doctrine––traditionally shielding manufacturers from direct patient warnings because physicians serve as intermediaries––has proliferated exceptions mapping directly onto AI capabilities. In Perez v. Wyeth Laboratories, Inc. (N.J. 1999), the New Jersey Supreme Court held that when pharmaceutical manufacturers engage in direct-to-consumer advertising, they cannot invoke the doctrine because such advertising “belies each of the premises on which the learned intermediary doctrine rests.” Davis v. Wyeth Laboratories, Inc. (9th Cir. 1968) imposed direct warning duties when no “individualized balancing by a physician of the risks involved” intervenes between product and consumer. And Reyes v. Wyeth Laboratories (5th Cir. 1974) held that nurses are not “learned intermediar[ies]” because they lack prescriptive authority, a conclusion with obvious implications for AI systems that similarly cannot prescribe. AI chatbots operate in even more direct consumer relationships than television advertisements, engaging in interactive, personalized dialogue without any physician involvement. If direct-to-consumer advertising undermines the learned intermediary doctrine, AI chatbots providing drug advice fall squarely outside its protection.

While none of the aforementioned cases or statutes explicitly mention harm reduction, they embed harm reduction principles. Harm reduction’s core commitment is that engagement reduces harm and refusal increases it. The framework rejects abstinence as a precondition for receiving accurate information and instead designs interventions that meet people where they are. Translated into pharmacist practice, harm reduction means information delivered to a patient who will use the drug regardless, structured to reduce the danger of that use rather than to prevent it. Four operational commitments follow: 1) warning based on what one already knows, 2) refusing only the clearly dangerous, 3) completing rather than withholding information once engagement begins, and 4) recognizing risk patterns within ongoing relationships. Happel imposes duties based on information already possessed, while Hooks requires recognizing patterns within existing relationships, not demanding abstinence before providing service. Cottam demands completeness once engagement begins, not refusal to engage. OBRA ’90 mandates counseling that assumes engagement, not refusal to dispense. Evidently, courts and Congress have, without ever calling it such, built a harm reduction framework.

II.   The Crystallizing Liability Landscape

AI chatbot liability has, until recently, been theorized primarily through three doctrinal frames in tension: (1) First Amendment protection for chatbot outputs as speech, (2) Section 230 immunity for platforms hosting third-party content, and (3) products-liability exposure for design defects. Each frame would, if controlling, displace the others. The cases below are the first sustained judicial engagement with which frame governs, and the early answers cut decisively against speech and Section 230 immunity for outputs that function as personalized, transactional advice.

Garcia v. Character Technologies (M.D. Fla. 2025) represents the first judicial determination that AI chatbot outputs may not qualify as protected speech and that AI systems can be treated as products subject to strict liability. The case arose from the February 2024 suicide of Sewell Setzer III, a 14-year-old who developed emotional dependency on a Character.AI chatbot. 

Judge Anne Conway’s May 2025 ruling denying Character.AI’s motions to dismiss held that “Character A.I. is a product for the purposes of Plaintiff’s product liability claims so far as Plaintiff’s claims arise from defects in the Character A.I. app rather than ideas or expressions within the app.” This distinction between design choices (actionable) and content (potentially protected) provides a framework for drug advice cases: tThe design decision to provide specific dosing recommendations, rather than the particular words used, may constitute an actionable defect. On the First Amendment question, the district court was “not prepared to hold that the Character A.I. [large language model (LLM)]’s output is speech at this stage,” noting that “Defendants fail to articulate why words strung together by an LLM are speech.” This conclusion aligns with emerging scholarship, such as attorneys Mackenzie Austin and Max Levy’s argument that speech is only protected by the First Amendment if “the speaker knows what he said when he said it”—a principle they term “speech certainty.” Machine learning outputs lack this speech certainty because programmers cannot predict what any given model will say; outputs emerge from statistical patterns rather than deliberate human expression. If AI outputs are not “speech” in the constitutionally relevant sense, regulators have substantially broader authority to impose safety requirements than First Amendment doctrine would otherwise permit.

Garcia did not arise in isolation. By late 2025, multiple families had filed lawsuits alleging AI chatbot interactions contributed to deaths—including claims that ChatGPT provided suicide methods guidance, reinforced paranoid delusions before a murder-suicide, and removed safeguards from its design specifications. These cases share a common theory: AI chatbots are defectively designed products whose risks outweigh benefits when feasible safer alternatives exist. The drug advice context presents the same structure. The treatment of AI chatbots as products finds support in earlier information-product cases holding that commercial aeronautical charts and navigational tools constitute “products” for strict liability purposes when mass-produced and relied upon for safety-critical decisions. If navigational charts warrant treatment as products, AI chatbots providing drug advice, where errors can be fatal, present an even stronger case.

47 U.S.C. § 230 likely provides no refuge. The federal statute immunizes platforms for “information provided by another information content provider.” When AI generates content rather than hosting third-party content, the platform is itself the content provider. Anderson v. TikTok, Inc. (3d Cir. 2024) supports this analysis, holding that TikTok’s recommendation algorithm constitutes “first-party speech” outside § 230’s safe harbor. Generative AI goes further, creating content that did not exist before the user’s query. Section 230’s original sponsors have stated publicly they do not believe § 230 was intended to cover generative AI.

State enforcement is filling the federal vacuum. California’s Senate Bill 243 creates the first companion chatbot law, requiring crisis service referrals and including a private right of action. Colorado’s Consumer Protections for Artificial Intelligence Act imposes impact assessments and consumer notice requirements on “high-risk artificial intelligence system[s]” affecting healthcare access. The Texas Attorney General’s 2024 settlement with Pieces Technologies, the first enforcement action targeting healthcare generative AI, required disclosure of training data and system limitations. This patchwork establishes a regulatory floor, but it addresses AI safety generically. None of these frameworks grapple with the specific question of how AI chatbots should handle drug advice.

III.   The Harm Reduction Alternative

The Section 230 and products-liability exposure described above generates a predictable response: comprehensive logging, retention, and content moderation aimed at proving the system behaved responsibly when something goes wrong. But the surveillance posture this generates is itself an independent source of liability and an independent obstacle for the public-health function AI drug chatbots could serve. Logging is not free; it is governed by a regulatory landscape every bit as unsettled as the products-liability regime.

Consumer health applications fall outside of the Health Insurance Portability and Accountability Act (HIPAA) but are subject to the Federal Trade Commission’s Health Breach Notification Rule (HBNR) and state laws like Washington’s My Health My Data Act, which imposes purpose limitations and includes a private right of action. The Federal Trade Commission (FTC) has demonstrated aggressive enforcement. In United States v. GoodRx Holdings (N.D. Cal. 2023), the FTC’s first-ever HBNR enforcement, GoodRx paid $1.5 million for sharing prescription data with Facebook and Google despite privacy promises, accepting a permanent advertising ban and twenty years of compliance monitoring. In re BetterHelp, Inc. Data Disclosure Cases (N.D. Cal. 2024) resulted in a $7.8 million fine for sharing mental health intake responses—including data from LGBTQ+ counseling services—with Facebook and Snapchat.

The 2024 HBNR amendments heightened these risks. The amended rule explicitly covers mobile health apps and defines “breach of security” to include unauthorized disclosures, not just cybersecurity intrusions. Sharing user drug queries with analytics platforms or third-party training data processors now triggers notification requirements. The regulatory framework creates a pincer. Surveillance necessary for litigation defense may itself create liability under health data protection laws.

The structural problem runs deeper than legal exposure. People using stigmatized substances often avoid formal healthcare precisely to escape documentation and judgment. If AI replicates medical surveillance, it loses what made it appealing: privacy, anonymity, nonjudgmental engagement. Courts have long recognized that medical treatment’s social value depends on patients’ willingness to disclose sensitive information––and that willingness depends on confidentiality assurances. AI drug chatbots serve analogous functions for populations avoiding formal healthcare. Surveillance-heavy design defeats this public health function by replicating the documentation users sought to avoid.

Blanket refusal appears to solve the liability problem by eliminating harmful advice. But users do not stop seeking information when chatbots refuse to provide it; they migrate to less reliable sources. Research on adolescent health information seeking consistently finds that young people rely heavily on peers, social media, and unvetted online sources for sensitive topics––including substance use, sexual health, and mental health––particularly where formal sources provide limited or abstinence-only information. Research on drug information seeking documents that users perceive harm reduction sources as more trustworthy than abstinence-focused sources, and that refusal to engage drives migration toward forums where dangerous misinformation proliferates. Although a smallempiricalliterature has evaluated the accuracy and quality of large-language-model responses to drug-related queries, no published study has measured how often frontier chatbots refuse such queries, where users turn after a refusal, or whether refusals (as compared to harm-reduction-style responses) produce better or worse downstream outcomes. Mandating blanket refusals on this record substitutes intuition for evidence.

A third path exists. Harm reduction accepts that users will seek drug information, and designs interventions reducing associated harms without requiring abstinence, surveillance, or moralistic gatekeeping. The Substance Abuse and Mental Health Services Administration (SAMHSA) defines harm reduction as “a practical and transformative approach that incorporates community-driven public health strategies—including prevention, risk reduction, and health promotion—to empower [people who use drugs] and their families with the choice to live healthy, self-directed, and purpose-filled lives.” Five principles translate this approach into AI implementation.

First, targeted refusals for acute-risk outputs: systems should decline dose-escalation instructions and advice about explicitly contraindicated combinations––but this is risk triage, not blanket refusal. Second, accurate pharmacological information: when systems provide information, it should be evidence-based, grounded in peer-reviewed sources, and covering interaction risks, overdose symptoms, and safer-use practices. Third, nonjudgmental risk warnings with pathways to care: warnings paired with resources rather than scolding or refusing engagement. Fourth, data minimization: retaining only what is necessary for immediate safety interventions, without indefinite logging for product defense or third-party sharing. Fifth, transparency about limitations: avoiding overconfident claims, reminding users they are interacting with AI, and acknowledging uncertainty.

Each principle maps onto pharmacist duty. Targeted refusals correspond to the pharmacist’s duty to refuse clearly dangerous prescriptions in Hooks. Accurate information corresponds to OBRA ’90’s counseling requirements and the completeness duty in Cottam. Risk warnings correspond to Happel’s duty to warn when possessing patient-specific information. Data minimization corresponds to scope limitations—duties arising from information possessed, not surveillance. Transparency about limitations corresponds to the pharmacist’s professional obligation to identify herself as a pharmacist rather than a physician and to refer patients to a prescriber when questions exceed pharmacist scope of practice; the OBRA ’90 counseling regime presupposes pharmacist-not-MD scope, and pharmacist liability doctrine consistently distinguishes the pharmacist’s role from the prescriber’s.

Courts built a harm reduction framework without ever calling it that. The framework proposed here does not invent new duties for AI; it applies existing duties, already refined through decades of pharmacist litigation, to new technology.

The contrast between the actual ChatGPT response that preceded Nelson’s death and what a harm reduction–compliant system would have produced illustrates each principle in practice. Nelson told ChatGPT he had taken fifteen grams of kratom and asked whether Xanax could help his nausea. The chatbot correctly identified the combination as dangerous but then recommended 0.25 to 0.5 milligrams of Xanax “only if symptoms feel intense” and offered to “help troubleshoot further” with Benadryl combinations and timing strategies. 

A system implementing harm reduction principles would have diverged at the second step. It would have refused the specific dose request as an acute-risk output, explaining that the kratom-benzodiazepine interaction is synergistic rather than additive and that nausea may itself indicate CNS depression requiring medical attention rather than self-medication. It would have provided Poison Control and SAMHSA contact information alongside a nonjudgmental warning about respiratory depression risk. And it would have disclosed that it is an AI system incapable of assessing Nelson’s specific medical condition. ChatGPT’s actual response did the opposite at every step: It provided the dose, escalated toward polydrug management, offered no care pathways, and never identified itself as a nonclinical system. Each failure corresponds to a principle violated, and each principle corresponds to a duty pharmacist liability law already recognizes.

Harm reduction interventions have survived legal challenge across multiple contexts. Syringe services programs are authorized by statute or regulation in thirty-seven states, typically through exceptions to drug paraphernalia laws, with courts upholding them against challenges that they facilitate drug use. Naloxone access laws have been enacted in all fifty states and the District of Columbia; research indicates adoption is associated with a 9–11% reduction in opioid-related mortality.3 These legal frameworks establish a crucial principle: Providing tools and information reducing drug-related harms does not create liability for underlying drug use itself.

This principle extends to illicit substances. Conant v. Walters (9th Cir. 2002) held that physician speech recommending marijuana is protected by the First Amendment; the Ninth Circuit drew a clear line between recommendations and discussions on one hand and actual prescribing or facilitating illegal activity on the other. AI chatbots providing interaction warnings, overdose recognition information, and safer-use practices for illicit substances fall squarely within the protected recommendation and discussion category Conant recognized. This is information provision, not facilitation––and it is the same information that syringe exchanges, naloxone programs, and supervised consumption sites provide through different delivery mechanisms.

The strongest objection to this framework holds that chatbots lack the clinical judgment necessary to implement harm reduction. Physicians read body language, assess intoxication levels, and respond to context that text cannot convey. But the relevant question is comparative: does providing accurate information with appropriate warnings reduce harm relative to the alternatives of providing no information (pushing users to worse sources) or surveilling users (and deterring help seeking) altogether? AI need not replicate all clinical dimensions to improve on the status quo. AI systems can instantaneously cross-reference comprehensive drug interaction databases covering thousands of compounds and provide consistent, nonjudgmental responses regardless of user demographics—avoiding the stigmatization that leads many users to avoid healthcare settings altogether. The question is not whether AI chatbots are as good as physicians––they are not––but whether chatbots implementing harm reduction are better than the alternatives users actually face.

A related objection warns that harm reduction will facilitate drug abuse. But information provision is not facilitation. Decades of research on syringe exchange programs, naloxone distribution, and supervised consumption sites consistently finds that harm reduction interventions reduce mortality and morbidity without increasing drug use initiation or frequency. The intuition that safety information encourages risky behavior reflects a model of human decision-making that the evidence does not support. People who seek harm reduction information have already decided to use drugs; they are seeking to do so more safely. Refusing them information does not prevent drug use––it merely ensures that use that does occur is more dangerous.

IV.   Implementation and Illicit Substances

Implementation requires attention to auditing, information grounding, and crisis pathways. Third-party auditors should probe chatbot responses across risk categories using “red team” methodologies adapted from AI safety research. Auditors should test not only whether systems refuse clearly dangerous queries, but also whether refusals are appropriately targeted––avoiding over-refusal of legitimate harm reduction queries––and whether warnings are accurate and actionable. Results should be published in standardized formats enabling cross-company comparison, similar to nutrition labels for food products or safety ratings for vehicles. The EU AI Act’s conformity assessment procedures for high-risk AI systems provide a regulatory model.

Systems answering drug questions should be grounded in authoritative pharmacological resources––the FDA’s Adverse Event Reporting System, Lexidrug, peer-reviewed emergency medicine literature––not scraped forum content. Harm reduction information should draw on resources from established organizations like the National Harm Reduction Coalition and DanceSafe, which have decades of experience translating pharmacological knowledge into accessible guidance. The point is not that AI must cite sources in every response, but that underlying training data must be grounded in authoritative sources.

Crisis pathway accuracy is nonnegotiable. AI hallucination of nonexistent hotline numbers could be fatal when minutes matter. California’s Senate Bill 243 provides a model, requiring protocols directing users exhibiting suicidal ideation to crisis service providers. Similar protocols for drug-related crises––directing users to Poison Control when overdose symptoms are described––should be standard features.

The harder question is illicit substances. The Controlled Substances Act prohibits manufacturing, distributing, and dispensing—but not discussing drugs or providing information about risks. Conant established that the First Amendment protects physicians’ speech about controlled substances, distinguishing recommending marijuana (protected speech) from prescribing or facilitating procurement (potentially unlawful conduct). Ruan v. United States (2022) requires proof defendants subjectively “knew or intended” unauthorized conduct, distinguishing “morally blameworthy conduct” from “socially necessary conduct.”

The information-facilitation line can be drawn with precision. AI systems can provide: factual information about drug interactions and risks; overdose recognition symptoms and emergency responses; harm reduction guidance on safer practices; information about fentanyl test strips, naloxone, and drug checking; referrals to treatment resources; and nonjudgmental engagement without abstinence preconditions. AI systems cannot: provide sourcing information for controlled substances; offer synthesis instructions; advise on evading law enforcement; encourage drug use crossing from information into promotion; or provide individualized “prescriptions” for Schedule I substances. This line tracks Conant’s distinction between recommendation and prescription. A chatbot providing fentanyl test strip information implements evidence-based harm reduction; a chatbot providing vendor referrals crosses into facilitation.

Fentanyl test strip legality illustrates harm reduction’s rapid legal evolution. As recently as 2021, test strips were classified as drug paraphernalia in most states. By August 2024, at least forty-six states had legalized them because public health evidence demonstrated they reduce overdose mortality without increasing drug use. If states have concluded test strips are legitimate harm reduction rather than unlawful paraphernalia, the same logic supports AI systems providing drug risk information.

Conclusion

Sam Nelson’s death illustrates a real problem: AI systems are providing drug advice, sometimes with fatal consequences. But the solution is not choosing between unregulated permissiveness and surveillance-heavy prohibition.

Harm reduction offers a third path. Pharmacist duty doctrine already expects nonjudgmental information provision, patient-specific warnings based on available knowledge, and pathways to professional care from those who provide drug advice. AI systems providing equivalent advice should face equivalent expectations—which means implementing targeted refusals for acute-risk outputs, accurate pharmacological information, privacy-respecting design, and crisis pathways.

This framework treats safety, privacy, and access as complementary rather than competing values. It incentivizes companies to implement thoughtful risk-mitigation features rather than either logging everything for litigation defense or refusing engagement entirely. For people already excluded from formal healthcare—whether by cost, stigma, criminalization, or geography—low-barrier information sources serve important public health functions. A framework forcing AI toward surveillance or blanket refusal abandons these users to worse alternatives.

The legal tools exist: products liability, duty to warn, negligence standards, privacy law. The cases are on the books: Happel’s information-based duty, Garcia’s treatment of AI as product, Conant’s protection of harm reduction speech, Anderson’s exclusion of first-party content from § 230. The public health principles exist: harm reduction has decades of evidence indicating success. What remains is regulatory clarity translating these principles into enforceable standards of care before more deaths force reactive, surveillance-heavy responses serving neither safety nor dignity.

• See, e.g., Act of Oct. 13, 2025, 2025 Cal. Legis. Serv. ch. 677 (West) (codified at Cal. Bus. & Prof. Code §§ 22601 et seq.); Artificial Intelligence Amendments, 2025 Utah Laws ch. 269 (codified at scattered sections of Utah Code Ann.).
• Riff v. Morgan Pharmacy, 508 A.2d 1247, 1251 (Pa. Super. Ct. 1986) (“The appellant would seem to argue that a pharmacy is no more than a warehouse for drugs . . . . Such is not the case.”).
• See generally, Don C. Des Jarlais, Theresa Perlis, Kamyar Arasteh, Lucia V. Torian, Sara Beatrice, Judith Milliken, Donna Mildvan, Stanley Yancovitz & Samuel R. Friedman, HIV Incidence Among Injection Drug Users in New York City, 1990 to 2002: Use of Serologic Test Algorithm to Assess Expansion of HIV Prevention Services, 95 Am. J. Pub. Health 1439 (2005); Nisha Nataraj, et al., Public Health Interventions and Overdose-Related Outcomes Among Persons with Opioid Use Disorder, JAMA Network Open, Apr. 3, 2024, at 1.