CS651 | Web Systems
  • outline
  • projects
  • syllabus

Example - Speech Coach (SPA)

 

 

Concept

Suppose a student team wants to create a speech coach app (SPA) where small videos can be uploaded and analyzed giving both a critique, improvement and practice options.

 

 

React SPA
↓
Node/Express Backend
↓
Upload short video
↓
Extract / access:
• video frames
• audio track
↓
Gemini direct multimodal analysis
↓
Structured observations
↓
Google ADK Agent Architecure
         ↓
        Coaching Critique / Improvement Suggestions / Practice Options
↓
Firestore

 

 

 

 

 

 

Agent Architecture:

 

The architecture is a hierarchical Google ADK multi-agent system built around an Orchestrator Agent. The application first sends each relevant section of the uploaded speech video to Gemini for direct multimodal analysis, producing structured observations about Speech Content, Vocal Delivery, and Body Language. Those Gemini analysis results become the input to the ADK workflow.

The Orchestrator Agent manages three specialized agents in sequence. The Critique Agent interprets the Gemini observations and presents clear, section-by-section critiques for Content, Vocal Delivery, and Body Language. The Improvement Agent consumes those critiques and converts them into specific, actionable recommendations for each category and video section. Finally, the Practice Agent uses both the critiques and improvement recommendations to create targeted exercises—for example, rewriting an opening, practicing pacing and pauses, or rehearsing eye contact and gestures. The Orchestrator combines the three outputs into the final coaching experience shown in the React App and can persist the analysis, recommendations, and exercises in Firestore for later comparison and progress tracking.

 

 

Diagram illustrating the architecture of a Speech Coach web application that combines Google Gemini with a Google ADK multi-agent system. A student uploads a speech video through a React Single Page Application (SPA), which communicates with a Node/Express backend responsible for authentication, file uploads, API orchestration, and data access. The backend sends the video to Gemini for direct multimodal analysis, producing structured results from two analyses: a Video Analysis (posture, eye contact, gestures, facial expressions, movement) and an Audio/Speech Analysis (speech content, pace and fluency, filler words, tone and clarity, pauses and emphasis, and an optional transcript). These structured results are stored in Firestore and also provided to a Google ADK multi-agent system.

Within the ADK system, an Orchestrating Agent coordinates three specialized agents. The Critique Agent receives Gemini’s section-by-section analysis and generates structured critiques for Speech Content, Vocal Delivery, and Body Language, identifying strengths, weaknesses, evidence, and severity. The Improvement Agent uses the critique report to produce concrete recommendations for improving each of those three areas across different sections of the speech. The Practice Agent combines the critique and improvement recommendations to generate personalized practice exercises, including specific activities, instructions, durations, and goals for Speech Content, Vocal Delivery, and Body Language. The Orchestrating Agent assembles these outputs into a final coaching package.

The final output is presented to the student as a Coaching Report containing stylized critiques, an improvement plan, a practice plan, and progress tracking. Firestore stores user profiles, uploaded videos and metadata, Gemini analysis results, critique reports, improvement plans, practice plans, and historical progress. A summary at the bottom explains that Gemini performs the direct multimodal analysis while the ADK multi-agent system transforms those observations into actionable coaching recommendations and practice exercises, with a table summarizing the responsibilities of the Orchestrating, Critique, Improvement, and Practice Agents.

 

 

 

NOTE: The initial Gemini calls perform direct multimodal analysis of the video/audio. The ADK agents subsequently use Gemini as their reasoning model while operating on those analysis results and invoking any tools provided to the agents.The initial Gemini calls perform direct multimodal analysis of the video/audio. The ADK agents subsequently use Gemini as their reasoning model while operating on those analysis results and invoking any tools provided to the agents.

NOTE: This is one example of an appropriate agent architecture. Your Project 2 agent architecture should be designed specifically for the purpose of your application and does not need to follow this architecture.

cs651:web systems

  • home
  • outline
  • projects
  • syllabus