PromptForward

Getting Started

Set up PromptForward and run your first prompt test in minutes.

Key Concepts

Before diving in, let's understand the core components of PromptForward:

Prompts

Prompts are the instructions you give to AI models. In PromptForward, prompts include system messages, user messages, and variables. Each time you save changes, a new version is created, allowing you to track and compare improvements over time.

LLM Providers

LLM Providers are your connections to AI services like OpenAI, Anthropic, Groq, or AWS Bedrock. You configure providers with your API credentials so PromptForward can execute prompts using various models.

Datasets

Datasets are collections of test cases. Each row contains input data (like user questions or chat messages) and optionally expected outputs. Datasets let you systematically test your prompts against many scenarios at once.

Evaluators

Evaluators automatically score AI responses. System evaluators use exact matching, contains matching, or regex. Judge LLM evaluators use another AI model to assess quality based on custom criteria like tone, accuracy, or completeness.

Test Runs

Test Runs execute your prompt against every item in a dataset, applying evaluators to each response. They provide detailed results showing success rates, individual scores, and help you identify areas for improvement.

API Keys

API Keys enable your applications to access PromptForward prompts programmatically. You can set granular permissions to control which prompts and providers each key can access.

Quick Start

1. Configure an LLM Provider

Before testing prompts, connect to an AI service:

  1. Navigate to LLM Providers in the sidebar
  2. Click + New Provider
  3. Enter a name (e.g., "Production OpenAI")
  4. Select your provider (OpenAI, Anthropic, Groq, or AWS Bedrock)
  5. Enter your API credentials
  6. Click Test Connection to verify
  7. Click Create Provider
LLM Provider creation modal

2. Create Your First Prompt

  1. Navigate to Prompts
  2. Click + New Prompt
  3. Enter a name and description
  4. Click Create Prompt
  5. Go to the Playground tab
  6. Select your provider and model
  7. Edit the system message with your instructions
  8. Use {{variable_name}} for dynamic inputs
  9. Click Run to test
  10. Click Save to create version 1
Prompt playground with test output

3. Create a Dataset

  1. Navigate to Datasets
  2. Click + New Dataset
  3. Choose to use an example, upload CSV, or start empty
  4. Enter a dataset name
  5. Click Create Dataset
Dataset creation modal with example preview

4. Run a Test

  1. Navigate to Test Runs
  2. Click + New Test
  3. Select your provider, model, prompt, and dataset
  4. Add evaluators to assess responses
  5. Toggle Quick Run for a 20-item test
  6. Click Run Test
Test run results detail page

5. Integrate with API

  1. Set your prompt version as Live in the Versions tab
  2. Navigate to API Keys
  3. Click + New API Key
  4. Add permissions for your prompt and provider
  5. Copy the API key and use it in your application
API key detail page with permissions

Next Steps

Explore the feature documentation to learn more: