In a first for the U.S., Utah is letting artificial intelligence — not a doctor — renew certain medical prescriptions. No human involved.

The state has launched a pilot program with health-tech startup Doctronic that allows an AI system to handle routine prescription renewals for patients with chronic conditions. The initiative, which kicked off quietly last month, is a high-stakes test of whether AI can safely take on one of health care’s most sensitive tasks and how far that could spread beyond one AI-friendly red state.

It also serves as an early check on how far policymakers and patients are willing to trust AI over trained doctors in decision-making. By inserting algorithms into one of medicine’s most fundamental relationships, Utah’s initiative could represent the first step in upending how care is delivered in the U.S.

That raises new questions about the safety of automating prescription refills, including how they should be regulated. So far, the Food and Drug Administration has not weighed in on Doctronic’s program. If the agency determines it has authority to regulate this use of AI, it could complicate or slow its expansion.

State officials and industry backers say relying more on artificial intelligence lowers costs, reduces medication lapses and improves access to care — while generating data that could shape AI policy beyond Utah.

Health care expenses keep climbing and clinicians — especially in rural areas — are stretched thin, said Margaret Busse, executive director of the Utah Department of Commerce. The state sees automating routine prescription renewals as a way to ease pressure on providers while lowering costs for patients, she said.

It’s also a way to “provide a pathway to innovation for entrepreneurs who are using AI in creative ways that may be bumping up against regulation,” she said.

But doctors’ groups warn that delegating some aspects of prescribing medication to AI could present new hazards.

In a statement, Dr. John Whyte, CEO and executive vice president at the American Medical Association, said: “While AI has limitless opportunity to transform medicine for the better, without physician input it also poses serious risks to patients and physicians alike.”

One concern is misuse or abuse, including the possibility that people struggling with addiction could try to game automated systems to obtain drugs inappropriately. Another concern is missing subtle clinical red flags or drug interactions that a doctor would catch.

“The company has to do that kind of trust building with their patients,” said Busse. “We want it to be done in such a way that people will trust that Utah is looking at this carefully and is not being cavalier about how we granted this regulatory mitigation. In a way it’s a risk for us as we do this.”

Read More: Artificial intelligence begins prescribing medications in Utah