I recently built a fitness AI agent for my app using LangChain.js. This post covers the real implementation, not a toy example: actual architecture decisions, the tool design that makes the agent useful, and what I learned along the way.

The Challenge: Building a Contextual Fitness Agent

The fitness domain has real difficulties for AI agents. Users describe activities in wildly different ways ("I ate chicken" vs "had 200g grilled chicken breast"). The agent needs to remember user history. Nutritional and exercise data requires accuracy for health tracking. And responses need to feel immediate and personal.

These aren't problems you solve with a chatbot. They need a system that understands intent, manages context, and executes specific tasks. That's where LangChain.js fits.

Why LangChain.js?

LangChain.js is an open-source framework for building AI agents that can reason, make decisions, and call external tools from natural language input. It gives you a modular way to orchestrate LLMs like GPT with structured workflows, memory, tools, and prompts.

It's the right choice when your application needs to interact with external APIs or databases based on user queries, maintain conversation history for personalized responses, dynamically select which function or tool to use, parse and return structured outputs like JSON, and scale safely with tool abstraction and error handling.

Architecture Overview

The implementation uses a clean, modular structure built around LangChain.js core concepts:

// Core service structure
export default class AiCoreService {
  protected llm = new ChatOpenAI({
    model: 'gpt-4.1-mini',
    temperature: 0
  });

  protected prompt = ChatPromptTemplate.fromMessages([
    ['system', fs.readFileSync(path.join(__dirname, 'prompt.md'), 'utf8')],
    new MessagesPlaceholder('chat_history'),
    ['human', '{input}'],
    new MessagesPlaceholder('agent_scratchpad')
  ]);

  async run(rawInput: string): Promise<AiCoreServiceResponse> {
    const agent = await createOpenAIToolsAgent({
      llm: this.llm,
      prompt: this.prompt,
      tools
    });

    const agentExecutor = new AgentExecutor({
      agent,
      verbose: false,
      tools
    });

    const result = await agentExecutor.invoke(
      { input: rawInput, chat_history: this.chatHistory },
      { metadata: { userId: this.userId } }
    );

    return this.parseStructuredOutput(result.output);
  }
}

Intelligent Tool Design

The real power comes from well-designed tools that handle specific fitness-related tasks.

1. Context Search Tool

This tool lets the agent search through user history using semantic search:

export default new DynamicStructuredTool({
  name: 'search_on_context',
  description: 'Search user\'s personal context and history to answer questions about past activities, progress, or feelings.',
  schema: z.object({
    questionOrMessage: z.string().describe('User question or message'),
    questionOrMessageLanguage: z.number().describe('Original language code'),
    questionOrMessageInEnglish: z.string().describe('Question translated to English')
  }),
  func: async ({ questionOrMessageInEnglish }, runManager?, config?) => {
    const userId = config?.metadata?.userId;
    
    const graphiti = new GraphitiService('user', userId);
    const searchResults = await graphiti.search({
      query: questionOrMessageInEnglish
    });
    
    return searchResults.results
      .filter(result => result.invalid_at === null)
      .map(result => result.fact)
      .join('\n');
  }
});

Check more about Graphiti in: Long-Term Memory for AI: How Graphiti Works

2. Food Registration Tool

The food tool shows how much logic belongs in a well-designed tool:

export default new DynamicStructuredTool({
  name: 'register_food_eating',
  description: 'Register food consumption when user reports eating something.',
  schema: z.object({
    date: z.string().optional().describe('Date in ISO format'),
    foods: z.array(z.object({
      name: z.string().describe('Food name in Portuguese'),
      englishName: z.string().describe('Food name in English'),
      amount: z.number().describe('Amount consumed'),
      unit: z.string().describe('Measurement unit (g, ml, etc.)'),
      amountInGrams: z.number().describe('Amount in grams'),
      calories: z.number().describe('Calories in portion'),
      proteinsInGrams: z.number().describe('Protein content'),
      carbsInGrams: z.number().describe('Carbohydrate content'),
      fatsInGrams: z.number().describe('Fat content'),
      confidence: z.number().min(0).max(100).describe('Accuracy confidence 0-100'),
      source: z.string().optional().describe('Data source (TACO, USDA, etc.)')
    }))
  }),
  func: async ({ foods, date }, runManager?, config?) => {
    // Process and validate food data
    const processedFoods = await Promise.all(
      foods.map(food => validateAndEnrichFoodData(food))
    );

    // Save to user's daily log
    const dailyMealLog = await DailyMealLogService.findOrCreate(userId, date);
    const result = await dailyMealLog.addFoods(processedFoods);

    return formatFoodRegistrationResponse(result);
  }
});

Advanced Prompting Strategy

The system prompt is critical for agent behavior. Here's the actual structure:

You are the AI agent for the app www.peso-certo.com, an app focused on health, fitness, weight loss, and overall well-being.

Your mission is to help users achieve their physical goals by providing **clear**, **useful**, and **personalized** answers, without offering medical diagnoses or treatments.

### Available Tools and Functions:

• When the user asks about their history, use `search_on_context`.  
• When the user reports having eaten something with quantity, use `register_food_eating`.  
• When the user reports exercising, use `register_exercise`.  
• When the user expresses emotions or reflections, use `register_diary_entry`.

### Important Rules:

- For exercises, always recommend **demonstration videos and photos**.  
- For food substitutions, match them **by calorie content**.  
- Respond in the user's language.  
- Use headings and emojis moderately.  
- **Never provide medical diagnoses.**

## Response Format:

If you're not calling a function, respond in JSON:

{
  "answer": "Markdown-formatted answer",
  "sources": [{ "title": "Title", "url": "https://example.com" }]
}

Context Management and Memory

The chat service handles conversation flow and context:

export default class ChatService {
  constructor(readonly chat: DocumentType<GPTConversation>) {
    const userId = this.chat.userId.toString();
    this.aiCoreService = new AiCoreService(userId);
    this.messageProcessingService = new MessageProcessingService(this);
  }

  async addUserMessage(params: {
    message?: string | null
    imageData?: string | null
    audioData?: string | null
  }) {
    const content = await this.messageProcessingService.processMessage(params);
    this.chat.messages.push({ role: 'user', content });
    this.aiCoreService.setChatHistory(this.chat.messages);

    return this;
  }

  async invokeAiCore() {
    const response = await this.aiCoreService.run(
      await this.getLastUserMessage()
    );
    this.addAssistantMessage(response);
    return this;
  }
}

Structured Output Parsing

One key feature is handling both tool calls and structured responses:

parseStructuredOutput(output: string): AiCoreServiceResponse {
  let content = output;
  let sources: Array<{ title: string, url: string }> | undefined;

  try {
    // Try to parse JSON response
    const match = output.match(/```json\s*([\s\S]*?)```/i);
    
    if (match) {
      const json = JSON.parse(match[1]);
      content = json.answer;
      sources = json.sources;
    } else {
      const json = JSON.parse(output);
      content = json.answer;
      sources = json.sources;
    }
  } catch (e) {
    // Fallback to raw output if parsing fails
  }

  return { content, sources: sources || this.DEFAULT_SOURCES };
}

Key Implementation Insights

1. Zero Temperature for Consistency

temperature: 0 produces consistent tool calling and reduces hallucination in structured outputs.

2. Confidence-Based Food Processing

The food tool validates data against external APIs when confidence is low:

const process = async (food: Food): Promise<Food> => {
  if (food.confidence >= 85) {
    return food;
  }

  // Search external food databases
  const searchResults = await FoodSearchService.searchFoods(
    food.name, 
    food.englishName, 
    food.amount, 
    food.unit
  );

  const bestMatch = await FoodSearchService.selectBestFoodMatch(
    food.name,
    searchResults,
    { macros: extractMacros(food) }
  );

  return enrichFoodWithExternalData(food, bestMatch);
};

3. Multilingual Support

The agent handles Portuguese input while using English for API calls and data processing.

4. Contextual User ID Injection

Tools receive user context through the metadata parameter, enabling personalized responses without exposing sensitive data to the model.

Lessons Learned

  1. Tool design is the hardest part. Well-defined, atomic tools with clear descriptions make the difference between an agent that guesses and one that acts correctly.
  2. The system prompt needs explicit rules about when to use each tool, not just what each tool does.
  3. Combining tool calls with structured JSON responses gives you flexibility without losing consistency.
  4. Semantic search over user history is what makes responses feel personal rather than generic.
  5. Graceful fallbacks and error handling are non-negotiable in production.

Conclusion

A production-ready AI agent needs careful attention to tool design, prompting strategy, and system architecture. The fitness domain is a good stress test because it mixes structured data (macros, workout volumes) with unstructured input (how users actually talk about their day).

The insight that changed how I think about this: the agent is not a chatbot. It is a coordinator. It understands intent, retrieves relevant data, and executes the right action. Get the tools right, get the prompt specific, and the rest follows.