Google Open-Sources FashionMind: Capable of Analyzing Outfits, Remembering Preferences, and Delivering Real-Time Recommendations

Google Open-Sources FashionMind: Capable of Analyzing Outfits, Remembering Preferences, and Delivering Real-Time Recommendations

2026-11-17 12:20

Introduction: Direct context injection via product catalog eliminates 6–16 seconds of voice round-trip overhead. What is FashionMind? FashionMind is Google’s open-source multimodal real-time voice fashion assistant, built on Google ADK and Gemini Live 2.5 Flash.

What is FashionMind?

FashionMind is Google’s open-source multimodal real-time voice fashion assistant, powered by Google ADK and Gemini Live 2.5 Flash. It enables users to view outfits through a camera, engage in natural voice conversations, retain user preferences across sessions, store precise factual data in relational databases, preserve style preferences in semantic memory, and extract only conclusions from raw audio-video inputs. The core design of FashionMind leverages direct product catalog injection into context and read-write separation, enabling seamless full-duplex conversation without any additional voice round-trips.

FashionMind

Key Features of FashionMind

  • Real-time Full-Duplex Voice Conversation: Built on native audio from Gemini Live, enabling continuous, uninterrupted dialogue with the user—no need to wait for turn-based responses.
  • Visual Outfit Evaluation: Real-time analysis of 1fps camera feed allows the system to “see” the user’s outfit and deliver natural commentary within the same voice turn.
  • Cross-Session Memory: Retains user schedules, wishlists, purchase history, and style preferences; no need to restate context upon subsequent launches.
  • Zero-Latency Product Recommendations: Product catalog pre-injected into context enables immediate frontend highlighting of corresponding cards when a product is mentioned—eliminating extra voice rounds.
  • Instant Barge-in (Interruption): User speech immediately halts ongoing audio playback; local detection triggers interruption without waiting for server-side VAD signal return.

Technical Principles of FashionMind

  • Real-Time Dual-Task Architecture: Backend runs as a FastAPI WebSocket service executing two concurrent asyncio tasks. The upstream task pushes browser-provided 16kHz PCM audio, images, and text into the LiveRequestQueue, while the downstream task iterates runner.run_live() events and forwards them to the frontend. Queues are managed via send_realtime() for continuous audio (detected via VAD speech boundaries) and send_content() for discrete image, text, and context injection (capable of triggering inference or interruption instantly), decoupling WebSocket handlers from ADK runner logic.
  • Context Injection at Session Start: User identity, product catalog (~300 tokens), and cross-session style memory are injected simultaneously at t=0ms via non-blocking send_content(). Catalog becomes available in milliseconds, while memory arrives in ~1.5 seconds. Since injection does not interrupt the real-time audio stream, the agent already holds complete context by the time the user finishes their first utterance—enabling an immediate response like “Do you have suits?” without requiring tool calls.
  • Catalog Direct Injection & Signal Tool Mode: Any tool call in voice applications typically incurs an additional 2 voice rounds (offer → confirm → result), consuming ~6–16 seconds. This project abandons the search_products() retrieval tool in favor of directly embedding the product catalog into the model context. recommend_products() is designed as a pure signal tool—no database lookup occurs; it generates detectable function_call events within the ADK event stream. The downstream task uses these signals to push highlighting instructions to the frontend, synchronizing voice narration with card highlighting.

digging GitHub

WeChat follow and reply “open source” to join the AI Open Source Project Community

How to Use FashionMind

  • Prerequisites: A GCP project with billing enabled, authenticated gcloud CLI, and locally installed Python 3.11+ and Node.js 18+.
  • One-Click Cloud Infrastructure Setup: Run ./setup_gcp.sh YOUR_PROJECT_ID us-central1; the script automatically creates a Cloud SQL PostgreSQL instance and Vertex AI Agent Engine, and generates the backend/.env configuration file.
  • Launch Backend Service: Enter the backend directory, create a virtual environment, install dependencies, then run alembic upgrade head (database migration), python seed_products.py (import product catalog), followed by uvicorn main:app --reload --port 8080 to start the service.
  • Configure Environment Variables: Copy backend/.env.example to backend/.env, fill in database connection string, GCP project ID, region, Agent Engine ID, and set USE_MEMORY_BANK=true to enable cross-session memory.
  • Start Frontend Application: Enter the frontend directory, run npm install and npm run dev, then open http://localhost:5173.
  • Begin Conversational Experience: After granting camera and microphone permissions in the browser, users can directly chat via voice with the fashion assistant and showcase outfits.

Core Advantages of FashionMind

  • Zero Additional Voice Rounds, Ultra-Low Latency: Pre-injected product catalog and signal-tool-based recommendations eliminate the 6–16 second silence caused by tool calls—product mentions trigger instant card highlighting.
  • Three-Tier Memory Separation Ensures Accuracy and Flexibility: Precise facts stored in relational SQL transactions, dynamic preferences asynchronously distilled into semantic memory—avoiding the risk of vector databases guessing exact answers based on similarity.
  • Read-Write Separation Without Blocking Audio Stream: Explicit commands written synchronously immediately, while preference summaries are processed silently by background coroutines—ensuring uninterrupted full-duplex conversation.
  • t=0ms Instant Mic Activation, Context Asynchronously Ready: Audio stream starts immediately; catalog and memory injected in parallel asynchronously—by the time the user finishes speaking, the agent has full contextual awareness.
  • Local Barge-in Enables Immediate Response: Browser-side RMS threshold detects speech onset and stops playback instantly—no waiting for server-side VAD signal return.

Project Repository

  • GitHub Repository: https://github.com/cuppibla/fashionmind

Competitive Comparison with Similar Projects

Dimension FashionMind Pinterest Assistant
Product Positioning Open-source reference architecture for developers—reusable vertical voice application engineering template Consumer-ready feature embedded within Pinterest platform—currently in beta phase
Voice Interaction Mode Full-duplex continuous dialogue: 16kHz continuous audio stream, wake-word-free, fully interruptible (barge-in) Button-based voice input: hold mic to speak (similar to sending voice messages), non-streaming, audio playback cannot be disabled
Visual Capabilities 1fps real-time camera feed continuously analyzes outfits, delivering comments within the same voice turn Multimodal "visual-first" model capable of generating outfit suggestions from posters or screen images—but not real-time video streaming
Memory Architecture Three-tier separation: precise facts in relational DB, style preferences asynchronously distilled into semantic memory, raw audio-video only retains conclusions Relies on Pinterest user behavior signals (saved Pins, browsing history) for personalization—no publicly documented memory tiering mechanism
Product Integration Self-hosted product catalog fully injected into context—product mention triggers immediate frontend card highlighting Recommends internal Pins—voice describes pairing and displays visual result
Latency Engineering Deeply optimized: asynchronous catalog/memory injection, signal tools eliminating round-trips, local barge-in, local session state No public details on latency optimization; voice output intentionally kept brief

Application Scenarios of FashionMind

  • Online Fashion Shopping Guidance: Customers describe needs via voice; the agent instantly recommends products based on catalog and purchase history, with corresponding cards highlighted in real-time on the frontend.
  • Personal Styling Consultant: Users activate the camera to show their daily outfit; the agent provides real-time feedback and improvement suggestions, remembering style preferences across sessions.
  • Occasion-Based Outfit Planning: Integrates with relational database-saved schedules (e.g., dinner party, client meeting) to recommend suitable attire in advance and add to wishlist.
  • Smart Wardrobe Management: Users query orally—“What outerwear have I bought?” or “What’s in my wishlist?”—the agent retrieves structured records precisely and manages purchase status.
  • Accessibility-Assisted Shopping: Visually impaired or mobility-limited users complete browsing, asking, and adding-to-cart workflows entirely via voice—no screen interaction required.

Source: AI Tools Collection

#AI Tools

Disclaimer: Contains third-party opinions, does not constitute financial advice

Share To
X
Telegram
WeChat
QQ
Link
Recommended Reading

Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

09-24
Step Code – The Open-Source Terminal Programming Agent by Jieyue Star

Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

09-02
Hy4 Preview Lightweight Version – Tencent Hunyuan's Open-Source Hy4 Quantized Compression Model

Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

09-02
Claude Fable 5.1 – Anthropic's Latest Flagship Large Model

Open-source Retool is here, drag-and-drop backend with 60+ components

08-19
Open-source Retool is here, drag-and-drop backend with 60+ components

Claude Opus 5 is Here: Half the Price Approaching Fable 5

07-25
Claude Opus 5 is Here: Half the Price Approaching Fable 5

Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

07-25
Tencent Open-Sources 260 Real-World Tasks to Test AI Workforce Capabilities

Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks

07-25
Meituan Open-Sources AI Exam Platform: 813 Minecraft Tasks