Google's rollout of AI overviews for Google Search in the U.S. has revealed problematic 'hallucinations' in responses, including incorrect facts and confusions between satire and journalism. Despite these issues, Google emphasizes that most AI overviews are accurate and improvements are ongoing. The situation highlights the challenges and potential risks of implementing AI in large-scale search operations.