CNBC Report Raises Concerns Over Anthropic, OpenAI Plan for AI Risk Evaluators

Anthropic and OpenAI have proposed using embedded AI evaluators as a tool to help manage the risk of artificial intelligence models causing catastrophic harm to society, according to a report published by CNBC.

The report describes the proposal as an emerging idea within the AI industry aimed at addressing safety concerns tied to increasingly capable AI systems. Under this approach, AI-based evaluators would be built into models or systems to help assess and flag potential risks before they escalate into significant harm.

However, CNBC's report indicates that the concept faces notable issues that warrant scrutiny. The report does not detail the full scope of these concerns in the available excerpt, but frames the proposal as one that, while intended to bolster safety, may itself introduce new risks or limitations that have not been fully resolved.

Anthropic and OpenAI are among the leading developers of advanced AI models and have both previously emphasized safety research as part of their public messaging on AI development. The companies have not been quoted directly in the available excerpt regarding the specifics of the evaluator proposal.

CNBC's report suggests that the broader AI industry and policymakers may need to weigh the benefits and drawbacks of relying on AI systems themselves to police other AI systems for catastrophic risk, a debate that is likely to continue as companies explore new safety frameworks.