“Implementing an AI Evaluation Framework in Utah,” from the Aspen Institute’s Policy Academy, offers recommendations for a standardized evaluation framework for OAIP’s partners. The academy has drafted a document that aligns with OAIP’s focus areas: the “Prosperity, Integrity and Innovation, Openness, Natural Resource Stewardship, and Respect for Culture and Values (PIONR) Framework.”
OAIP was established in July to shape AI policy in the state. The new office builds on previous AI work in Utah, including the 2018 establishment of a Center of Excellence in AI and the 2023 generative AI policy.
Housed within the Utah Department of Commerce, the OAIP is focused externally on society rather than internally on state government, according to Utah CIO Alan Fuller.
“What the [OAIP] is focused on is helping the community around AI questions,” Fuller said. For example, if a company provides a tool using AI to deliver a service like mental health counseling, OAIP aims to answer questions around the rules that apply to that tool and who would be liable if it offers bad advice. “It’s both a lab and a policy place.”
OAIP, which is working to create a regulatory framework for AI, has been seeking input from technology stakeholders in the community to help guide the future of AI in Utah. The office was initially seeking input on specific solutions, Aspen Policy Academy fellow Ayodele Odubela said, but the Aspen Policy Academy fellows identified a need for more transparency around the evaluation system within OAIP’s regulatory mitigation program.
The Aspen Policy Academy, a part of the Aspen Institute that aims to support nonpartisan policy development, executed this project to support OAIP’s standardization of evaluation criteria. OAIP should publicize both the participation criteria and a running list of participants in its AI Learning Lab, according to academy recommendations. The project to create the framework was done as part of the 2024 Science and Technology Policy Fellowship, an Aspen Tech Policy Hub program, and released this month.
The OAIP has a webpage highlighting agreements between state agencies and external partners; one is currently listed. The creation of the page is a result of recommendations made in the framework, Odubela said.
In the process of creating this framework, Odubela said it became clear that people wanted to engage with state government about the way AI is being used, but there were barriers to doing so. As such, part of the work from the Aspen Policy Academy involved communicating directly with people who lived in Utah, and assessing their awareness of OAIP’s work.
A key focus of the initiative was to improve transparency for constituents, but the framework also helps companies that want to partner with the state identify areas they may have overlooked, Odubela said. This could include areas like social bias risks.
The recommendations from the Aspen Policy Academy are intended to increase transparency as AI innovation happens, and to help Utah serve as a model in responsible AI development.
Vendors looking to build AI tools and solutions can work proactively with OAIP to understand the rules and expectations from Utah government, Fuller said, so they can advance AI responsibly and meet state policies: “So, the idea is, we want to promote innovation, but we also need to have guardrails around that innovation.”
Federal changes in AI regulation under the new administration are shaping the broader AI legislative landscape. Because the current administration is repealing certain AI safeguards that were implemented under the prior administration, Odubela said there is a greater push to support legislation and other initiatives at the state and local levels focused on transparency and building trust, to address gaps that may exist at the federal level.
For the Aspen Policy Academy, the work is not only focused on this framework or on the state of Utah, she said — but on work that can resonate with every city and state. The ultimate goal is for people impacted by state AI tools to influence their development — especially, she emphasized, as residents cannot always opt out of technologies used by government.
This framework can be a starting point for other state and local governments to think more locally and consider the values that will guide their AI governance and implementations, Odubela noted.
The OAIP has the state CIO’s support, he said: “I think they’ll continue to be an influence for both innovation as well as the appropriate uses of AI in the community."
