Evaluator

Evaluate content

Assess content quality using customizable evaluation metrics and scoring criteria. Create objective evaluation frameworks with numeric scoring to measure performance across multiple dimensions.

Toolsevaluator

Usage

01Add the block to your workflow and connect it to the upstream step.
02Configure any required credentials or tokens in the inputs.
03Fill in required inputs and optional parameters for the run.
04Run a test execution, inspect outputs, and iterate before deploying.
05Deploy the evaluator block with monitoring enabled in production.

Inputs (UI)

Evaluation Metrics

eval-input

Layout: full

Content

short-input

Placeholder: Enter the content to evaluate

Layout: full

Model

dropdown

Layout: half

Options: Dynamic options

API Key

short-input

Placeholder: Enter API key or use $ENV_VAR — managed via Model Settings

Layout: full

Condition: model = ["gpt-4o","gpt-4o-mini","gpt-4.1","gpt-4.1-mini","gpt-5","gpt-5-mini","gpt-5-nano","gpt-5.2","gpt-5.4","o3-mini","o4-mini","o3","o1-mini","o1-preview","claude-opus-4-6","claude-sonnet-4-6","claude-opus-4-5-20251101","claude-sonnet-4-5-20250929","claude-opus-4-1-20250805","claude-sonnet-4-20250514","claude-opus-4-20250514","claude-haiku-4-5-20251001","gemini-3.1-pro-preview","gemini-3-flash-preview","gemini-3.1-flash-lite-preview","gemini-2.5-pro","gemini-2.5-flash","gemini-2.5-flash-lite","gemini-2.0-flash","gemini-2.0-flash-lite","deepseek-chat","deepseek-reasoner","grok-4-latest","grok-3","grok-3-mini","cerebras/llama-3.3-70b","llama-4-scout-17b-16e-instruct","llama-4-maverick-17b-128e-instruct","deepseek-r1-distill-llama-70b","qwen-qwq-32b","llama-3.3-70b-versatile"]

System Prompt

code

Layout: full

Hidden by default

Inputs (API)

metrics

json

Required

Array of metrics to evaluate against

model

string

Required

apiKey

string

Required

content

string

Required

Outputs

Primary response type:

{
  "content": "string",
  "model": "string",
  "tokens": "json"
}

Conditional output based on: metrics

When empty: {"content":"string","model":"string","tokens":"json"} | When filled: "json"

Tool Access

openai_chatanthropic_chatgoogle_chatxai_chatdeepseek_chatdeepseek_reasoner