What is Google Cloud’s generative AI evaluation service?

As part of Clarify, AWS offers enterprises a feature, dubbed FMEvalwhich is an open-source LLM evaluation library to help data scientists and ML engineers evaluate LLMs before deciding to use it for a specific use case.

“FMEval provides the ability to perform evaluations for both LLM model endpoints or the endpoint for a generative AI service as a whole. FMEval helps in measuring evaluation dimensions such as accuracy, robustness, bias, toxicity, and factual knowledge for any LLM,” the cloud service provider wrote in a blog post.

Enterprises can use EMEval to evaluate LLMs hosted on either AWS or third-party platforms, such as ChatGPTHuggingFace, and LangChain, it added.