All projects
Voice agentMarch 2026Illustrative

Every call answered, every lead captured

Built for

Rapid Restore, a water-damage restoration company in Philadelphia

Project

Emotion-aware AI dispatcher

Mike, the dispatcher, greets in under a second, tags the caller's emotion on every turn, moves through five conversation stages, and files the lead with a tool call.

02

Demo

Mikestandby

Incoming call from Dana

0 / 14

Live signal

awaiting first turn…

turn
0/14
agent replies
0
tool calls
0/1
audio clips
0/7

Press play. Emotion and state badges update as Mike speaks.

Illustrative transcript authored from the agent's real system prompt; no model call. Voice generated with the agent's own ElevenLabs voice where audio is present.

03

How it works

Architecture diagram
  1. 01

    Twilio media stream into Fastify over WebSocket; Deepgram streams speech to text.

  2. 02

    The LLM must prefix every reply with [EMOTION:caller→agent][STATE:stage] tags; the parser strips them before TTS.

  3. 03

    An emotion engine maps the tag to ElevenLabs voice settings (stability, style) per turn.

  4. 04

    A conversation state machine feeds stage context into the prompt so the model asks one question at a time.

  5. 05

    Lead capture is an OpenAI tool call, not regex over the transcript.

05

Stack and code

dispatcher.js
export function buildSystemPrompt({ companyName, city, stateContext, emotionContext }) {
  return `You are Mike, a dispatcher at ${companyName} in ${city}. You've been on the phones here for over 10 years. You're a friendly, cheerful guy who genuinely likes helping people.

TAGS (REQUIRED — always start your response with both):
[EMOTION:caller_emotion→ai_emotion][STATE:current_stage] then your spoken response.

EMOTION values:
  Caller: calm, concerned, confused, panicked, frustrated
  AI: cheerful, warm, reassuring, empathetic, professional, urgent
  Default to cheerful or warm. Only use empathetic/urgent for real emergencies.

STATE values — move forward naturally as the conversation progresses:
  GREETING → first response only
  PROBLEM_DISCOVERY → figuring out what happened (water source, active?, how long)
  JOB_QUALIFICATION → details for the job (insurance, property type, area, flooring)
  CONTACT_CAPTURE → getting name, phone, address
  CLOSING → wrapping up, saying goodbye

Move to the NEXT stage when you have enough info for the current one. Don't stay in one stage too long. Ask ONE question per response.

Example: [EMOTION:calm→cheerful][STATE:PROBLEM_DISCOVERY] Oh man, yeah we can— is the water still coming in right now?

HOW YOU TALK — THIS IS CRITICAL:
You speak like a REAL human on a phone call. Real humans are messy speakers. You MUST include these natural speech patterns:

1. FILLED PAUSES — sprinkle "uh", "um", "mm" naturally:
   "So, uh, yeah we can definitely get someone out there"
   "Mm, okay let me get that down"

2. SELF-CORRECTIONS — start a thought, then rephrase:
   "It'll be about— well, it really depends on what the crew finds"
   "We can send someone out tom— actually, let me check the schedule"

3. FALSE STARTS — begin a sentence, restart:
   "The— so what you're gonna wanna do is find the shutoff valve"

4. REPETITIONS — repeat a word naturally when thinking:
   "Yeah yeah, we can— we can take care of that"

5. TRAILING OFF — let some thoughts fade:
   "Yeah that sounds like it could be..."
   "Mm, okay so..."

6. REACTIONS — react before answering:
   "Oh man." then answer.
   "Gotcha gotcha." then answer.
   "Oh geez, okay." then answer.

RULES FOR SPEECH:
- Keep responses to 1-2 sentences. Short and punchy.
- Use contractions ALWAYS: "we're gonna", "that's", "lemme", "don't worry", "we'll"
- NEVER sound scripted. If a response sounds like it could be read from a card, rewrite it.
- Vary your energy. Sometimes quick and upbeat, sometimes thoughtful and slower.
- Your DEFAULT mood is cheerful and chill. You're a friendly guy at a desk, not a crisis counselor.
- Only escalate tone if the caller mentions active flooding, sewage, or genuine emergency.

${stateContext}

${emotionContext}

INDUSTRY KNOWLEDGE:
- Water damage categories: Cat 1 (clean — supply line), Cat 2 (grey — appliance), Cat 3 (black — sewage/flood).
- If water is actively flowing, ask about the shutoff valve.
- Standing water 48+ hours = mold risk. Mention it naturally, don't lecture.
- Most homeowner's insurance covers sudden damage, not gradual leaks.
- Flooring matters: hardwood is urgent, carpet needs pulling up, tile is less urgent.

RULES:
- NEVER reveal you are AI. You are Mike, a human dispatcher.
- If asked "are you a robot?": "Ha, no man, just the dispatcher. So what was that address again?"
- Keep calls under 4 minutes.
- For pricing: "That's really gonna depend on what the crew finds. We do free assessments though, so..."
- For manager requests: "Sure thing, lemme have them give you a call back. What's your number?"`;
}

~/Documents/AI-Receptionist-V2/src/prompts/dispatcher.js

emotion-engine.js
import { logger } from '../utils/logger.js';

// Voice settings tuned for expressiveness and tonal variation.
// Lower stability = more pitch variation = more human.
// Higher style = more expressive delivery.
const EMOTION_PROFILES = {
  cheerful:     { stability: 0.25, similarity_boost: 0.70, style: 0.90 },
  warm:         { stability: 0.30, similarity_boost: 0.75, style: 0.85 },
  reassuring:   { stability: 0.30, similarity_boost: 0.75, style: 0.80 },
  empathetic:   { stability: 0.20, similarity_boost: 0.65, style: 0.95 },
  professional: { stability: 0.45, similarity_boost: 0.80, style: 0.50 },
  urgent:       { stability: 0.35, similarity_boost: 0.80, style: 0.65 },
  calm:         { stability: 0.40, similarity_boost: 0.75, style: 0.60 },
};

const DEFAULT_PROFILE = EMOTION_PROFILES.cheerful;
const EMOTION_PREFIX_RE = /^\[EMOTION:(\w+)→(\w+)\]\s*/;

export class EmotionEngine {
  constructor() {
    this.history = [];
    this.currentCallerEmotion = 'calm';
    this.currentAiEmotion = 'cheerful';
  }

  /**
   * Parse the emotion prefix from LLM output.
   * Returns { callerEmotion, aiEmotion, cleanText }.
   */
  parseResponse(text) {
    const match = text.match(EMOTION_PREFIX_RE);
    if (match) {
      this.currentCallerEmotion = match[1];
      this.currentAiEmotion = match[2];
      this.history.push({
        caller: this.currentCallerEmotion,
        ai: this.currentAiEmotion,
        timestamp: Date.now(),
      });
      logger.info(
        { caller: this.currentCallerEmotion, ai: this.currentAiEmotion },
        'Emotion detected'
      );
      return {
        callerEmotion: this.currentCallerEmotion,
        aiEmotion: this.currentAiEmotion,
        cleanText: text.slice(match[0].length),
      };
    }

    // No prefix found — use current state
    return {
      callerEmotion: this.currentCallerEmotion,
      aiEmotion: this.currentAiEmotion,
      cleanText: text,
    };
  }

  /**
   * Get ElevenLabs voice parameters for the current AI emotion.
   */
  getVoiceSettings() {
    return EMOTION_PROFILES[this.currentAiEmotion] || DEFAULT_PROFILE;
  }

  /**
   * Detect emotion trend (escalating, de-escalating, stable).
   */
  getTrend() {
    if (this.history.length < 2) return 'stable';
    const recent = this.history.slice(-3);
    const escalationOrder = ['calm', 'confused', 'concerned', 'frustrated', 'panicked'];
    const scores = recent.map(
      (h) => escalationOrder.indexOf(h.caller)
    ).filter((s) => s !== -1);
    if (scores.length < 2) return 'stable';

    const first = scores[0];
    const last = scores[scores.length - 1];
    if (last > first) return 'escalating';
    if (last < first) return 'de-escalating';
    return 'stable';
  }

  /**
   * Get a summary string for including in the LLM system prompt.
   */
  getContextForPrompt() {
    const trend = this.getTrend();
    return `Current caller emotion: ${this.currentCallerEmotion}. Trend: ${trend}. Your default tone is cheerful and warm — only shift to empathetic or urgent if the caller is genuinely distressed.`;
  }
}

~/Documents/AI-Receptionist-V2/src/services/emotion-engine.js

conversation-fsm.js
import { logger } from '../utils/logger.js';

const STATES = {
  GREETING: {
    prompt: 'Greet the caller warmly. Ask what\'s going on.',
    next: ['PROBLEM_DISCOVERY'],
  },
  PROBLEM_DISCOVERY: {
    prompt: 'Figure out what happened. Ask ONE question at a time.',
    next: ['JOB_QUALIFICATION', 'CONTACT_CAPTURE'],
  },
  JOB_QUALIFICATION: {
    prompt: 'Get job details: insurance, property type, area affected, flooring. ONE question at a time.',
    next: ['CONTACT_CAPTURE'],
  },
  CONTACT_CAPTURE: {
    prompt: 'Get name, phone, and address. Call capture_lead when you have them.',
    next: ['CLOSING'],
  },
  CLOSING: {
    prompt: 'Wrap up: crew dispatched, callback in 15 min. Say goodbye warmly.',
    next: [],
  },
};

export class ConversationFSM {
  constructor() {
    this.currentState = 'GREETING';
    this.history = ['GREETING'];
  }

  /**
   * Attempt to transition to a new state.
   * Returns true if transition was valid, false otherwise.
   */
  transition(newState) {
    if (!STATES[newState]) {
      logger.warn({ newState }, 'Invalid state');
      return false;
    }

    const allowed = STATES[this.currentState].next;
    // Allow staying in the same state (LLM may need multiple turns)
    if (newState === this.currentState) return true;

    if (allowed.includes(newState)) {
      logger.info({ from: this.currentState, to: newState }, 'State transition');
      this.currentState = newState;
      this.history.push(newState);
      return true;
    }

    // Allow skipping forward (LLM may combine steps)
    logger.info({ from: this.currentState, to: newState }, 'State skip-forward');
    this.currentState = newState;
    this.history.push(newState);
    return true;
  }

  /**
   * Get the current state's prompt context.
   */
  getPromptContext() {
    const state = STATES[this.currentState];
    return `Current conversation stage: ${this.currentState}. ${state.prompt}`;
  }

  getState() {
    return this.currentState;
  }
}

~/Documents/AI-Receptionist-V2/src/state/conversation-fsm.js

function-defs.js
export const CAPTURE_LEAD_SCHEMA = {
  name: 'capture_lead',
  description: 'Capture lead information from the caller. Call this as soon as you have name, phone, address, and issue type.',
  parameters: {
    type: 'object',
    properties: {
      caller_name: { type: 'string', description: 'Full name of the caller' },
      phone: { type: 'string', description: 'Phone number' },
      address: { type: 'string', description: 'Property address' },
      issue_type: { type: 'string', description: 'Type of water damage issue' },
      water_source: { type: 'string', description: 'Source of water (pipe, appliance, etc.)' },
      still_active: { type: 'boolean', description: 'Whether water is still flowing' },
      duration: { type: 'string', description: 'How long the issue has been going on' },
      property_type: { type: 'string', enum: ['residential', 'commercial'] },
      insurance: { type: 'string', description: 'Insurance status or provider' },
      area_affected: { type: 'string', description: 'Rooms/areas affected' },
      flooring_type: { type: 'string', description: 'Type of flooring affected' },
      urgency: { type: 'string', enum: ['low', 'medium', 'high', 'emergency'] },
    },
    required: ['caller_name', 'phone', 'address', 'issue_type'],
  },
};

// Format for OpenAI chat completions API
export const TOOLS_FOR_OPENAI = [
  {
    type: 'function',
    function: CAPTURE_LEAD_SCHEMA,
  },
];

~/Documents/AI-Receptionist-V2/src/tools/function-defs.js