Example Sets

Example sets are collections of ground truth input/output pairs used to evaluate pipeline performance. They define the expected behavior of a pipeline and serve as the foundation for running evaluations.

Key Concepts

Example Sets - Collections that define input/output schemas and contain multiple examples
Examples - Individual ground truth pairs within a set
Configurations - Versioned snapshots of the example set schema (append-only)

Example Set Structure

An example set consists of:

Name & Description - Human-readable identifiers
Inputs Schema - Schema defining what inputs examples will have
Outputs Schema - Schema defining expected outputs
Configuration - Pre-filled values and settings
Status - active or archived

Creating an Example Set

Create an example set

curl -X POST https://api.catalyzed.ai/example-sets \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "teamId": "ZkoDMyjZZsXo4VAO_nJLk",
    "name": "Document Summarization Examples",
    "description": "Ground truth examples for document summarization pipeline",
    "inputsSchema": {
      "files": [],
      "datasets": [],
      "dataInputs": [
        {
          "id": "document",
          "name": "Document Text",
          "type": "string",
          "required": true
        }
      ]
    },
    "outputsSchema": {
      "files": [],
      "datasets": [],
      "dataInputs": [
        {
          "id": "summary",
          "name": "Summary",
          "type": "string",
          "required": true
        }
      ]
    }
  }'

const response = await fetch("https://api.catalyzed.ai/example-sets", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${apiToken}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    teamId: "ZkoDMyjZZsXo4VAO_nJLk",
    name: "Document Summarization Examples",
    description: "Ground truth examples for document summarization pipeline",
    inputsSchema: {
      files: [],
      datasets: [],
      dataInputs: [
        {
          id: "document",
          name: "Document Text",
          type: "string",
          required: true,
        },
      ],
    },
    outputsSchema: {
      files: [],
      datasets: [],
      dataInputs: [
        {
          id: "summary",
          name: "Summary",
          type: "string",
          required: true,
        },
      ],
    },
  }),
});
const exampleSet = await response.json();

response = requests.post(
    "https://api.catalyzed.ai/example-sets",
    headers={"Authorization": f"Bearer {api_token}"},
    json={
        "teamId": "ZkoDMyjZZsXo4VAO_nJLk",
        "name": "Document Summarization Examples",
        "description": "Ground truth examples for document summarization pipeline",
        "inputsSchema": {
            "files": [],
            "datasets": [],
            "dataInputs": [
                {
                    "id": "document",
                    "name": "Document Text",
                    "type": "string",
                    "required": True
                }
            ]
        },
        "outputsSchema": {
            "files": [],
            "datasets": [],
            "dataInputs": [
                {
                    "id": "summary",
                    "name": "Summary",
                    "type": "string",
                    "required": True
                }
            ]
        }
    }
)
example_set = response.json()

Response:

{
  "exampleSetId": "KjR8I6rHBms3W4Qfa2-FN",
  "teamId": "ZkoDMyjZZsXo4VAO_nJLk",
  "name": "Document Summarization Examples",
  "description": "Ground truth examples for document summarization pipeline",
  "activeConfigurationId": "CfgABC123...",
  "inputsSchema": { ... },
  "outputsSchema": { ... },
  "configuration": { ... },
  "status": "active",
  "createdAt": "2024-01-15T10:30:00Z",
  "updatedAt": "2024-01-15T10:30:00Z",
  "createdBy": "usr_abc123"
}

Listing Example Sets

List example sets

curl "https://api.catalyzed.ai/example-sets?teamIds=ZkoDMyjZZsXo4VAO_nJLk" \
  -H "Authorization: Bearer $API_TOKEN"

const response = await fetch(
  "https://api.catalyzed.ai/example-sets?teamIds=ZkoDMyjZZsXo4VAO_nJLk",
  { headers: { Authorization: `Bearer ${apiToken}` } }
);
const { exampleSets } = await response.json();

response = requests.get(
    "https://api.catalyzed.ai/example-sets",
    params={"teamIds": "ZkoDMyjZZsXo4VAO_nJLk"},
    headers={"Authorization": f"Bearer {api_token}"}
)
example_sets = response.json()["exampleSets"]

Query Parameters

Parameter	Type	Description
`exampleSetIds`	string	Comma-separated list of IDs
`teamIds`	string	Comma-separated team IDs
`name`	string	Partial match filter
`status`	string	`active` or `archived`
`page`	number	Page number (1-indexed)
`pageSize`	number	Results per page (1-100)
`orderBy`	string	`createdAt`, `name`, `updatedAt`, `status`
`orderDirection`	string	`asc` or `desc`

Getting an Example Set

Get example set details

curl https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN"

const response = await fetch(
  "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN",
  { headers: { Authorization: `Bearer ${apiToken}` } }
);
const exampleSet = await response.json();

response = requests.get(
    "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN",
    headers={"Authorization": f"Bearer {api_token}"}
)
example_set = response.json()

Updating an Example Set

Update metadata like name and description:

Update example set

curl -X PUT https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Document Summarization Examples v2",
    "description": "Updated ground truth examples"
  }'

await fetch("https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN", {
  method: "PUT",
  headers: {
    Authorization: `Bearer ${apiToken}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    name: "Document Summarization Examples v2",
    description: "Updated ground truth examples",
  }),
});

requests.put(
    "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN",
    headers={"Authorization": f"Bearer {api_token}"},
    json={
        "name": "Document Summarization Examples v2",
        "description": "Updated ground truth examples"
    }
)

Configuration Versioning

Example sets use append-only configuration versioning. When you need to update the schema, create a new configuration version:

Create new configuration version

curl -X POST https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/configurations \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "inputsSchema": {
      "files": [],
      "datasets": [],
      "dataInputs": [
        {
          "id": "document",
          "name": "Document Text",
          "type": "string",
          "required": true
        },
        {
          "id": "maxLength",
          "name": "Max Summary Length",
          "type": "number",
          "required": false
        }
      ]
    },
    "outputsSchema": {
      "files": [],
      "datasets": [],
      "dataInputs": [
        {
          "id": "summary",
          "name": "Summary",
          "type": "string",
          "required": true
        }
      ]
    },
    "configuration": {},
    "changeReason": "Added optional maxLength input parameter"
  }'

const response = await fetch(
  "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/configurations",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      inputsSchema: { /* ... */ },
      outputsSchema: { /* ... */ },
      configuration: {},
      changeReason: "Added optional maxLength input parameter",
    }),
  }
);
const config = await response.json();

response = requests.post(
    "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/configurations",
    headers={"Authorization": f"Bearer {api_token}"},
    json={
        "inputsSchema": { ... },
        "outputsSchema": { ... },
        "configuration": {},
        "changeReason": "Added optional maxLength input parameter"
    }
)
config = response.json()

Listing Configuration History

curl "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/configurations" \
  -H "Authorization: Bearer $API_TOKEN"

Rolling Back to a Previous Configuration

To roll back, set activeConfigurationId to a previous version:

curl -X PUT https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "activeConfigurationId": "CfgPreviousVersion..."
  }'

Example Set Status

Status	Description
`active`	Example set can be used for evaluations
`archived`	Example set is disabled

Archiving an Example Set

Archive example set

curl -X POST https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/archive \
  -H "Authorization: Bearer $API_TOKEN"

await fetch("https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/archive", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiToken}` },
});

requests.post(
    "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/archive",
    headers={"Authorization": f"Bearer {api_token}"}
)

Reactivating an Archived Example Set

curl -X POST https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN/reactivate \
  -H "Authorization: Bearer $API_TOKEN"

Deleting an Example Set

Delete example set

curl -X DELETE https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN"

await fetch("https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN", {
  method: "DELETE",
  headers: { Authorization: `Bearer ${apiToken}` },
});

requests.delete(
    "https://api.catalyzed.ai/example-sets/KjR8I6rHBms3W4Qfa2-FN",
    headers={"Authorization": f"Bearer {api_token}"}
)

Example Set Properties

Field	Type	Description
`exampleSetId`	string	Unique identifier
`teamId`	string	Team that owns this example set
`name`	string	Human-readable name (1-255 chars)
`description`	string	Optional description
`activeConfigurationId`	string	Currently active configuration version
`inputsSchema`	object	Schema for example inputs
`outputsSchema`	object	Schema for expected outputs
`configuration`	object	Pre-filled configuration values
`status`	string	`active` or `archived`
`createdAt`	timestamp	Creation time
`updatedAt`	timestamp	Last modification time
`createdBy`	string	User who created this example set

Next Steps

Examples - Add individual examples to your example set
Evaluations - Run evaluations using your example sets
Evaluation Workflow Guide - End-to-end evaluation tutorial