I'm Samuel Fajreldines

I am a specialist in the entire JavaScript and TypeScript ecosystem (including Node.js, React, Angular and Vue.js)

I am expert in AI and in creating AI integrated solutions

I am expert in DevOps and Serverless Architecture (AWS, Google Cloud and Azure)

I am expert in PHP and its frameworks (such as Codeigniter and Laravel).

Samuel Fajreldines

I am a specialist in the entire JavaScript and TypeScript ecosystem.

I am expert in AI and in creating AI integrated solutions.

I am expert in DevOps and Serverless Architecture

I am expert in PHP and its frameworks.

+55 (51) 99226-5039 samuelfajreldines@gmail.com
Home Page

Building a Fitness AI Agent with LangChain.js: A Real-World Implementation

Building intelligent AI agents that can understand natural language and perform complex tasks has become increasingly important in modern app development. Recently, I built a comprehensive fitness AI agent for my app using LangChain.js that can understand user input, make intelligent decisions, and execute appropriate actions. This post explores the real implementation, architecture decisions, and lessons learned.

The Challenge: Building a Contextual Fitness Agent

The fitness domain presents unique challenges for AI agents:

  • Natural Language Complexity: Users describe activities in varied ways ("I ate chicken" vs "had 200g grilled chicken breast")
  • Context Awareness: The agent needs to remember user history and preferences
  • Data Precision: Nutritional and exercise data requires accuracy for health tracking
  • Multi-modal Input: Supporting text, voice, and potentially image inputs
  • Real-time Feedback: Providing immediate, personalized responses

Overcoming these challenges requires more than just a chatbot—it demands a system capable of understanding intent, managing context, and executing specific tasks. That’s where LangChain.js comes in.

Why LangChain.js?

LangChain.js is a powerful open-source framework designed to build AI agents that can reason, make decisions, and call external tools based on natural language input. It provides a modular way to orchestrate large language models (LLMs) like GPT with structured workflows, memory, tools, and prompts—all essential for building intelligent, context-aware applications.

LangChain shines when your application needs to:

  • Interact with external APIs or databases based on user queries
  • Maintain conversation history for personalized responses
  • Dynamically decide which function or tool to use
  • Parse and return structured outputs, like JSON
  • Scale safely into production with tool abstraction and error handling

Architecture Overview

The implementation uses a clean, modular architecture built around LangChain.js core concepts:

// Core service structure
export default class AiCoreService {
  protected llm = new ChatOpenAI({
    model: 'gpt-4.1-mini',
    temperature: 0
  });

  protected prompt = ChatPromptTemplate.fromMessages([
    ['system', fs.readFileSync(path.join(__dirname, 'prompt.md'), 'utf8')],
    new MessagesPlaceholder('chat_history'),
    ['human', '{input}'],
    new MessagesPlaceholder('agent_scratchpad')
  ]);

  async run(rawInput: string): Promise<AiCoreServiceResponse> {
    const agent = await createOpenAIToolsAgent({
      llm: this.llm,
      prompt: this.prompt,
      tools
    });

    const agentExecutor = new AgentExecutor({
      agent,
      verbose: false,
      tools
    });

    const result = await agentExecutor.invoke(
      { input: rawInput, chat_history: this.chatHistory },
      { metadata: { userId: this.userId } }
    );

    return this.parseStructuredOutput(result.output);
  }
}

Intelligent Tool Design

The real power comes from well-designed tools that handle specific fitness-related tasks:

1. Context Search Tool

This tool enables the agent to search through user history using semantic search:

export default new DynamicStructuredTool({
  name: 'search_on_context',
  description: 'Search user\'s personal context and history to answer questions about past activities, progress, or feelings.',
  schema: z.object({
    questionOrMessage: z.string().describe('User question or message'),
    questionOrMessageLanguage: z.number().describe('Original language code'),
    questionOrMessageInEnglish: z.string().describe('Question translated to English')
  }),
  func: async ({ questionOrMessageInEnglish }, runManager?, config?) => {
    const userId = config?.metadata?.userId;
    
    const graphiti = new GraphitiService('user', userId);
    const searchResults = await graphiti.search({
      query: questionOrMessageInEnglish
    });
    
    return searchResults.results
      .filter(result => result.invalid_at === null)
      .map(result => result.fact)
      .join('\n');
  }
});

Check more about Graphiti in: https://www.samuelfaj.com/posts/2025-07-14-222718/index.html

2. Food Registration Tool

The food tool demonstrates sophisticated data processing and validation:

export default new DynamicStructuredTool({
  name: 'register_food_eating',
  description: 'Register food consumption when user reports eating something.',
  schema: z.object({
    date: z.string().optional().describe('Date in ISO format'),
    foods: z.array(z.object({
      name: z.string().describe('Food name in Portuguese'),
      englishName: z.string().describe('Food name in English'),
      amount: z.number().describe('Amount consumed'),
      unit: z.string().describe('Measurement unit (g, ml, etc.)'),
      amountInGrams: z.number().describe('Amount in grams'),
      calories: z.number().describe('Calories in portion'),
      proteinsInGrams: z.number().describe('Protein content'),
      carbsInGrams: z.number().describe('Carbohydrate content'),
      fatsInGrams: z.number().describe('Fat content'),
      confidence: z.number().min(0).max(100).describe('Accuracy confidence 0-100'),
      source: z.string().optional().describe('Data source (TACO, USDA, etc.)')
    }))
  }),
  func: async ({ foods, date }, runManager?, config?) => {
    // Process and validate food data
    const processedFoods = await Promise.all(
      foods.map(food => validateAndEnrichFoodData(food))
    );

    // Save to user's daily log
    const dailyMealLog = await DailyMealLogService.findOrCreate(userId, date);
    const result = await dailyMealLog.addFoods(processedFoods);

    return formatFoodRegistrationResponse(result);
  }
});

Advanced Prompting Strategy

The system prompt is crucial for agent behavior. Here's the actual prompt structure used:

You are the AI agent for the app www.peso-certo.com, an app focused on health, fitness, weight loss, and overall well-being.

Your mission is to help users achieve their physical goals by providing **clear**, **useful**, and **personalized** answers, without offering medical diagnoses or treatments.

### Available Tools and Functions:

• When the user asks about their history, use `search_on_context`.  
• When the user reports having eaten something with quantity, use `register_food_eating`.  
• When the user reports exercising, use `register_exercise`.  
• When the user expresses emotions or reflections, use `register_diary_entry`.

### Important Rules:

- For exercises, always recommend **demonstration videos and photos**.  
- For food substitutions, match them **by calorie content**.  
- Respond in the user's language.  
- Use headings and emojis moderately.  
- **Never provide medical diagnoses.**

## Response Format:

If you're not calling a function, respond in JSON:

{
  "answer": "Markdown-formatted answer",
  "sources": [{ "title": "Title", "url": "https://example.com" }]
}

Context Management and Memory

The chat service handles conversation flow and context:

export default class ChatService {
  constructor(readonly chat: DocumentType<GPTConversation>) {
    const userId = this.chat.userId.toString();
    this.aiCoreService = new AiCoreService(userId);
    this.messageProcessingService = new MessageProcessingService(this);
  }

  async addUserMessage(params: {
    message?: string | null
    imageData?: string | null
    audioData?: string | null
  }) {
    const content = await this.messageProcessingService.processMessage(params);
    this.chat.messages.push({ role: 'user', content });
    this.aiCoreService.setChatHistory(this.chat.messages);

    return this;
  }

  async invokeAiCore() {
    const response = await this.aiCoreService.run(
      await this.getLastUserMessage()
    );
    this.addAssistantMessage(response);
    return this;
  }
}

Structured Output Parsing

One key feature is handling both tool calls and structured responses:

parseStructuredOutput(output: string): AiCoreServiceResponse {
  let content = output;
  let sources: Array<{ title: string, url: string }> | undefined;

  try {
    // Try to parse JSON response
    const match = output.match(/```json\s*([\s\S]*?)```/i);
    
    if (match) {
      const json = JSON.parse(match[1]);
      content = json.answer;
      sources = json.sources;
    } else {
      const json = JSON.parse(output);
      content = json.answer;
      sources = json.sources;
    }
  } catch (e) {
    // Fallback to raw output if parsing fails
  }

  return { content, sources: sources || this.DEFAULT_SOURCES };
}

Key Implementation Insights

1. Zero Temperature for Consistency

Using temperature: 0 ensures consistent tool calling and reduces hallucination in structured outputs.

2. Confidence-Based Food Processing

The food tool implements intelligent data validation, falling back to external APIs when confidence is low:

const process = async (food: Food): Promise<Food> => {
  if (food.confidence >= 85) {
    return food;
  }

  // Search external food databases
  const searchResults = await FoodSearchService.searchFoods(
    food.name, 
    food.englishName, 
    food.amount, 
    food.unit
  );

  const bestMatch = await FoodSearchService.selectBestFoodMatch(
    food.name,
    searchResults,
    { macros: extractMacros(food) }
  );

  return enrichFoodWithExternalData(food, bestMatch);
};

3. Multilingual Support

The agent handles Portuguese input while using English for API calls and data processing.

4. Contextual User ID Injection

Tools receive user context through the metadata parameter, enabling personalized responses without exposing sensitive data to the model.

Lessons Learned

  1. Tool Design is Critical: Well-defined, atomic tools with clear descriptions lead to better agent decisions
  2. Prompting Matters: Specific instructions about when to use tools dramatically improve accuracy
  3. Structured Output: Combining tool calls with structured JSON responses provides flexibility
  4. Context is King: Semantic search over user history enables truly personalized experiences
  5. Error Recovery: Graceful fallbacks and error handling are essential for production

Conclusion

Building a production-ready AI agent with LangChain.js requires careful attention to tool design, prompting strategy, and system architecture. The fitness domain provides an excellent case study for complex, multi-modal AI interactions.

The key to success lies in understanding that the agent is not just a chatbot—it's an intelligent coordinator that can understand intent, access relevant data, and execute appropriate actions. By combining LangChain.js's powerful orchestration capabilities with domain-specific tools and careful prompt engineering, we can create AI experiences that truly understand and assist users in their fitness journey.

The full implementation demonstrates that with the right architecture, AI agents can handle real-world complexity while maintaining reliability, performance, and user satisfaction.

Resume

Experience

  • SecurityScoreCard

    Nov. 2023 - Present

    New York, United States

    Senior Software Engineer

    I joined SecurityScorecard, a leading organization with over 400 employees, as a Senior Full Stack Software Engineer. My role spans across developing new systems, maintaining and refactoring legacy solutions, and ensuring they meet the company's high standards of performance, scalability, and reliability.

    I work across the entire stack, contributing to both frontend and backend development while also collaborating directly on infrastructure-related tasks, leveraging cloud computing technologies to optimize and scale our systems. This broad scope of responsibilities allows me to ensure seamless integration between user-facing applications and underlying systems architecture.

    Additionally, I collaborate closely with diverse teams across the organization, aligning technical implementation with strategic business objectives. Through my work, I aim to deliver innovative and robust solutions that enhance SecurityScorecard's offerings and support its mission to provide world-class cybersecurity insights.

    Technologies Used:

    Node.js Terraform React Typescript AWS Playwright and Cypress