The Global Network Initiative (GNI) – a multistakeholder membership organization focused on the intersections of tech, privacy, and freedom of expression – recently convened members and external experts for the second in a two-part learning series on government demands to technology companies. Building on the first discussion on “reverse searches”, a growing type of government demand for user information that raises important questions about privacy and freedom of expression, this call focused on government demands involving hosted AI models, including requests for user data and restrictions on AI-generated content.
The call brought together perspectives from GNI’s members – including civil society, academia, online platforms, and telecommunications companies – and external experts to discuss the types of information AI companies collect, how existing legal frameworks apply to government requests for this information, and the challenges companies face when responding to demands to restrict AI content or functionality. This piece highlights key learnings and areas of discussion from the conversation; it does not intend to be a comprehensive review of the issues.
Matt O’Shaughnessy, senior technologist at CDT’s AI Governance Lab, discussed how the information collected by AI providers can differ from that held by more traditional online platforms. For example, in addition to account information, AI companies may retain user prompts, conversation histories, and information inferred about users based on their interactions with AI systems. These conversations can include highly personal information, sometimes more than users would share through traditional search engines or other online services.
He also noted the additional risks associated with AI agents that are given access to users’ data, applications, and services. For example, AI tools connected to email, workplace messaging platforms, or web browsers may have access to private communications, browsing history, screenshots, and login credentials. Users may not always be aware of the amount or types of information these systems can access.
These new types of AI services increase the overall amount of data collected about users, and likely the amount of data that can be requested by governments. However, Matt noted that not all newly accessible data will necessarily be available to governments. What can be disclosed depends on whether companies collect and retain the data, and whether it is stored in a way that can be linked to a particular user. Therefore, company decisions around data retention and product design play a significant role in determining what information may be subject to government requests. Participants noted there are real tradeoffs between data retention for improved AI accuracy, personalization, and safety efforts, and data minimization for privacy. The discussion also touched on the potential role of privacy-enhancing technologies, such as encryption and private processing, in managing these tradeoffs, while noting that the level of protection depends on how these technologies are implemented.
Shreya Tewari, GNI’s policy lead, described ongoing research that GNI is doing on genAI data and foreshadowed a report that GNI will be publishing examining how that data is being accessed through legal processes, including government demands. She returned to the issue of “reverse searches,” discussed during the first call. She noted that in the AI context, law enforcement can seek to identify users who have entered particular prompts into a chatbot, rather than requesting information about an already identified individual. This is illustrated in a recent U.S. case, analyzed by the scholar Riana Pfefferkorn, where the government sought to unmask a ChatGPT user based on their specific prompts to a chatbot, which the user had shared in an anonymous chat with an undercover FBI officer. This specific request was about child safety. It was appropriately scoped and helped to solve a disturbing case. However, it should also be seen as a technique that could be used in inappropriate ways, if appropriate safeguards are not in place. Because AI interactions often involve extended conversations, these types of requests could implicate particularly sensitive user information.
Richard Salgado, a former Google lawyer, prosecutor and longtime legal expert on these issues who teaches at both Stanford and Harvard Law Schools, explored how existing legal frameworks govern compelled government access to information held by AI providers. In the United States, the Fourth Amendment and statutes such as the Stored Communications Act provide protections for information held by providers, including those offering artificial intelligence services. The Supreme Court, he said, recently made quite clear that the Fourth Amendment applies to much of the information such providers hold for or about users, and typically a warrant is required to force providers to disclose that information.
He noted that as AI systems increasingly mediate our communications, activities, and decisions, and gain insight into our cognitive processes, and subconscious minds, the Fourth Amendment’s particularity requirement for warrants will take on new significance. Identifying information responsive to a warrant, he said, may require providers, on behalf of the government, to search vast amounts of sensitive information drawn from user interactions, connected devices, a world of sensors, and outside data stores. It may be impossible to confine the search to the information particularly described in the warrant, especially when the system’s analysis cannot be explained or reconstructed. The possession, custody, or control standard and the rules governing authentication of records will come under similar strain when a provider’s systems derive information that may not exist as identifiable records and that no person at the provider can meaningfully access or explain. AI systems’ window into our thoughts may also revive the venerable “private papers” doctrine and raise the more fundamental question whether some records of our cognitive processes lie beyond the reach of even an otherwise valid warrant.
Participants also discussed transparency reporting in this context, which GNI previously explored in more depth in an earlier learning call. Although companies such as OpenAI and Anthropic have begun publishing information about government demands for user data, the number of demands they disclose remains relatively low compared to requests received by established online platforms. Participants suggested that these demands could increase as AI services become more widely used and as law enforcement becomes more familiar with the types of information available.
The call also looked at another type of government demand: requests to restrict AI-generated content or functionality.
Participants explained that content restriction requests to social media platforms and search engines generally identify a particular piece of content, such as a specific post, account, or URL, that a company can remove or restrict in a specific country, frequently known as geoblocking or georestricting. The GNI Implementation Guidelines call on companies to interpret government demands narrowly, minimize restrictions on freedom of expression, and limit compliance to the relevant jurisdiction where possible. This approach is complicated by the way AI systems work, where content is generated in response to user prompts and the same or similar information can be produced in different ways and there is often no way to uniquely identify the content described in a government request. This poses significant questions for the field about what rights-respecting company responses to these kinds of government demands might look like in the AI age.
The discussion identified several types of requests that governments might make, including asking companies to provide or restrict certain government-mandated information to targeted users, change specific AI-generated responses, prevent certain information from being generated, or restrict particular features of an AI product.
Some requests may be appropriate and rights-respecting, such as correcting inaccurate information like an incorrect phone number for a public service. Others may raise freedom of expression concerns, particularly where governments seek to prevent AI systems from discussing politically sensitive topics or require them to provide an approved version of events.
Participants also raised questions about how companies could implement these restrictions narrowly. Unlike removing a specific URL, restricting an AI response may require changes that affect a wider range of prompts or users. It may also be difficult to limit these changes to a particular jurisdiction.
These issues create challenges for the current practices that underpin industry-wide transparency reporting and user notice, as it may not always be clear to users when a government request has influenced the information an AI system provides.
The discussion highlighted several areas where further work is needed, including understanding how companies manage and retain AI user data, how existing legal protections apply to new types of technologies, and how companies can respond to AI-related government demands or restrictions in maximally rights-respecting ways.
As AI tools become more integrated into everyday activities, companies almost certainly will face a wider range of government demands involving both user information and generated content. The discussion pointed to the need for continued engagement across sectors to better understand these requests and consider how existing human rights safeguards can apply to AI systems.