Synthesis Runs

Synthesis runs analyze signals and execution history to propose pipeline configuration improvements. An LLM examines feedback patterns and suggests changes that address identified issues.

Key Concepts

Synthesis Run - An LLM-driven analysis that proposes configuration changes
Signals - Input feedback (comments, ratings, evaluations) that inform the synthesis
Diagnosis - Analysis of patterns and issues found in signals
Changes - Proposed modifications to pipeline configuration
Apply/Reject - Human decision on whether to implement proposed changes

Synthesis Lifecycle

pending → generating → generated → approved
                                 ↘ rejected
pending → failed

Status	Description
`pending`	Job queued, waiting to start
`generating`	LLM analysis in progress
`generated`	Proposal ready for review
`approved`	Changes applied to pipeline
`rejected`	Proposal declined
`failed`	Generation error

Initiating a Synthesis

Initiate synthesis

curl -X POST https://api.catalyzed.ai/pipelines/EMbMEFLyUWEgvnhMWXVVa/synthesize \
  -H "Authorization: Bearer $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "handlerType": "apg_v1"
  }'

const response = await fetch(
  "https://api.catalyzed.ai/pipelines/EMbMEFLyUWEgvnhMWXVVa/synthesize",
  {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      handlerType: "apg_v1",
    }),
  }
);
const synthesisRun = await response.json();
console.log(synthesisRun.synthesisRunId); // "SynR8I6rHBms3W4Qfa2-FN"

response = requests.post(
    "https://api.catalyzed.ai/pipelines/EMbMEFLyUWEgvnhMWXVVa/synthesize",
    headers={"Authorization": f"Bearer {api_token}"},
    json={"handlerType": "apg_v1"}
)
synthesis_run = response.json()
print(synthesis_run["synthesisRunId"])  # "SynR8I6rHBms3W4Qfa2-FN"

The response returns 202 Accepted since synthesis runs asynchronously:

{
  "synthesisRunId": "SynR8I6rHBms3W4Qfa2-FN",
  "pipelineId": "EMbMEFLyUWEgvnhMWXVVa",
  "teamId": "ZkoDMyjZZsXo4VAO_nJLk",
  "status": "pending",
  "handlerType": "apg_v1",
  "configurationBeforeId": "CfgABC123...",
  "configurationAfterId": null,
  "createdAt": "2024-01-15T10:30:00Z",
  "createdBy": "usr_abc123"
}

Handler Types

Type	Description
`apg_v1`	Analyze-Plan-Generate multi-step pipeline (DIAGNOSE → PLAN → AUTHOR → APPLY)

Getting Synthesis Status

Get synthesis run details

curl https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN"

const response = await fetch(
  "https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN",
  { headers: { Authorization: `Bearer ${apiToken}` } }
);
const synthesisRun = await response.json();

response = requests.get(
    "https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN",
    headers={"Authorization": f"Bearer {api_token}"}
)
synthesis_run = response.json()

Completed synthesis response:

{
  "synthesisRunId": "SynR8I6rHBms3W4Qfa2-FN",
  "pipelineId": "EMbMEFLyUWEgvnhMWXVVa",
  "teamId": "ZkoDMyjZZsXo4VAO_nJLk",
  "status": "generated",
  "handlerType": "apg_v1",
  "configurationBeforeId": "CfgABC123...",
  "configurationAfterId": "CfgDEF456...",
  "diagnosis": {
    "summary": "Users report summaries are missing key financial metrics and lack consistent formatting.",
    "findings": [
      {
        "id": "1",
        "type": "pain_point",
        "description": "Summaries frequently omit revenue growth percentages",
        "signalIds": ["SigXYZ1...", "SigXYZ2..."],
        "severity": "high"
      },
      {
        "id": "2",
        "type": "opportunity",
        "description": "Users prefer bullet-point format for key metrics",
        "signalIds": ["SigXYZ3..."],
        "severity": "medium"
      }
    ]
  },
  "changes": [
    {
      "id": "1",
      "type": "modify",
      "section": "configuration",
      "category": "dataInputs",
      "slotId": "system_prompt",
      "description": "Updated system prompt to emphasize financial metrics",
      "rationale": "Addresses finding #1 - ensures revenue percentages are included",
      "findingRefs": ["1"]
    },
    {
      "id": "2",
      "type": "add",
      "section": "configuration",
      "category": "dataInputs",
      "slotId": "output_format",
      "description": "Added output format instruction for bullet points",
      "rationale": "Addresses finding #2 - standardizes output format",
      "findingRefs": ["2"]
    }
  ],
  "structuredWarnings": [
    {
      "type": "assumption",
      "message": "Assumed all documents contain financial metrics",
      "changeRefs": ["1"]
    }
  ],
  "confidence": "high",
  "signalIds": ["SigXYZ1...", "SigXYZ2...", "SigXYZ3..."],
  "startedAt": "2024-01-15T10:30:02Z",
  "completedAt": "2024-01-15T10:35:00Z",
  "createdAt": "2024-01-15T10:30:00Z",
  "createdBy": "usr_abc123"
}

Polling for Completion

async function waitForSynthesis(synthesisRunId: string) {
  while (true) {
    const response = await fetch(
      `https://api.catalyzed.ai/synthesis-runs/${synthesisRunId}`,
      { headers: { Authorization: `Bearer ${apiToken}` } }
    );
    const run = await response.json();

    if (run.status === "generated") {
      console.log(`Diagnosis: ${run.diagnosis.summary}`);
      console.log(`Changes proposed: ${run.changes.length}`);
      return run;
    }
    if (run.status === "failed") {
      throw new Error(run.errorMessage);
    }
    if (["approved", "rejected"].includes(run.status)) {
      return run; // Already decided
    }

    await new Promise(r => setTimeout(r, 2000)); // Poll every 2 seconds
  }
}

Applying a Synthesis

Once status is generated, review the proposed changes and apply them:

Apply synthesis

curl -X POST https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/apply \
  -H "Authorization: Bearer $API_TOKEN"

await fetch("https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/apply", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiToken}` },
});

requests.post(
    "https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/apply",
    headers={"Authorization": f"Bearer {api_token}"}
)

When applied:

Status changes to approved
configurationAfterId becomes the pipeline’s new activeConfigurationId
appliedAt and appliedBy are recorded

Rejecting a Synthesis

If the proposed changes are not appropriate, reject them:

Reject synthesis

curl -X POST https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/reject \
  -H "Authorization: Bearer $API_TOKEN"

await fetch("https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/reject", {
  method: "POST",
  headers: { Authorization: `Bearer ${apiToken}` },
});

requests.post(
    "https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN/reject",
    headers={"Authorization": f"Bearer {api_token}"}
)

Listing Synthesis Runs

List synthesis runs

curl "https://api.catalyzed.ai/synthesis-runs?pipelineIds=EMbMEFLyUWEgvnhMWXVVa" \
  -H "Authorization: Bearer $API_TOKEN"

const response = await fetch(
  "https://api.catalyzed.ai/synthesis-runs?pipelineIds=EMbMEFLyUWEgvnhMWXVVa",
  { headers: { Authorization: `Bearer ${apiToken}` } }
);
const { synthesisRuns } = await response.json();

response = requests.get(
    "https://api.catalyzed.ai/synthesis-runs",
    params={"pipelineIds": "EMbMEFLyUWEgvnhMWXVVa"},
    headers={"Authorization": f"Bearer {api_token}"}
)
synthesis_runs = response.json()["synthesisRuns"]

Query Parameters

Parameter	Type	Description
`synthesisRunIds`	string	Comma-separated list of IDs
`pipelineIds`	string	Comma-separated pipeline IDs
`teamIds`	string	Comma-separated team IDs
`statuses`	string	Comma-separated: `pending`, `generating`, `generated`, `approved`, `rejected`, `failed`
`handlerTypes`	string	Comma-separated handler types
`createdBy`	string	Filter by creator
`createdAfter`	date	Filter by creation date
`createdBefore`	date	Filter by creation date
`page`	number	Page number (1-indexed)
`pageSize`	number	Results per page (1-100)
`orderBy`	string	`createdAt`, `appliedAt`, `status`
`orderDirection`	string	`asc` or `desc`

Deleting a Synthesis Run

Delete synthesis run

curl -X DELETE https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN \
  -H "Authorization: Bearer $API_TOKEN"

await fetch("https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN", {
  method: "DELETE",
  headers: { Authorization: `Bearer ${apiToken}` },
});

requests.delete(
    "https://api.catalyzed.ai/synthesis-runs/SynR8I6rHBms3W4Qfa2-FN",
    headers={"Authorization": f"Bearer {api_token}"}
)

Diagnosis Structure

The diagnosis field contains the LLM’s analysis:

Field	Type	Description
`summary`	string	1-2 sentence overview of findings
`findings`	array	Individual issues identified

Finding Types

Type	Description
`pain_point`	User-reported problem
`error_pattern`	Recurring error or failure
`opportunity`	Potential improvement
`positive`	What’s working well

Finding Severity

Severity	Description
`high`	Critical issue affecting many users
`medium`	Moderate impact or frequency
`low`	Minor issue or edge case

Changes Structure

The changes array describes proposed modifications:

Field	Type	Description
`id`	string	Change identifier
`type`	string	`add`, `modify`, or `remove`
`section`	string	`inputsSchema`, `outputsSchema`, or `configuration`
`category`	string	`files`, `datasets`, or `dataInputs`
`slotId`	string	ID of the slot being changed
`description`	string	Human-readable description
`rationale`	string	Why this change addresses issues
`findingRefs`	array	Finding IDs this addresses

Warnings Structure

The structuredWarnings array contains caveats:

Field	Type	Description
`type`	string	`breaking_change`, `assumption`, `tradeoff`, `needs_review`
`message`	string	Warning description
`changeRefs`	array	Related change IDs

Synthesis Properties

Field	Type	Description
`synthesisRunId`	string	Unique identifier
`pipelineId`	string	Pipeline being improved
`teamId`	string	Team that owns this synthesis
`status`	string	Current status
`handlerType`	string	Synthesis handler used
`configurationBeforeId`	string	Original configuration
`configurationAfterId`	string	Proposed configuration (when generated)
`diagnosis`	object	Analysis of signals
`changes`	array	Proposed modifications
`structuredWarnings`	array	Caveats and warnings
`confidence`	string	`low`, `medium`, or `high`
`signalIds`	array	Signals analyzed
`errorMessage`	string	Error details (if failed)
`startedAt`	timestamp	When generation started
`completedAt`	timestamp	When generation finished
`appliedAt`	timestamp	When approved and applied
`appliedBy`	string	User who approved
`createdAt`	timestamp	When synthesis was created
`createdBy`	string	User who initiated

Best Practices

Accumulate signals first - Gather enough feedback before running synthesis
Review changes carefully - Understand the rationale before applying
Check warnings - Pay attention to assumptions and tradeoffs
Test after applying - Run evaluations to verify improvements
Iterate - Synthesis is iterative; collect more signals and run again

Next Steps

Signals - Capture feedback to inform synthesis
Evaluations - Measure improvements after synthesis
Feedback Loops Guide - End-to-end workflow