CS 663 — Project 2: Intelligent Mobile Vision System

***** This is a GROUP Project *******

- everyone in the group will get the same score with the exception of points for peer revewing (TBD)
- only if necessary will I disband a group and then everyone in the group is responsible for creating their own project 2.
- Note except for serious issues like harassment, etc. I will not disband a group after Oct. 25.

             (See Canvas for points + Peer Reviews)

 

In this project, your group will design and implement a mobile computer vision application that solves a meaningful real-world problem.

Your application must use the camera of a mobile device to observe the real world and must use multiple AI/computer vision models working together as an integrated system to interpret visual input and provide useful information, assistance, or actions to the user.

This is not simply a model-training project and it is not a Colab-only project

You are building a complete Intelligent Mobile Vision System.

General System Concept

REAL WORLD
↓
MOBILE DEVICE / CAMERA
↓
IMAGE / VIDEO PROCESSING
↓
MULTIPLE AI / COMPUTER VISION MODELS
↓
REASONING + APPLICATION LOGIC
↓
USER OUTPUT / ACTION

 

Your particular architecture must make sense and may be different. Models may operate sequentially, in parallel, conditionally, or iteratively.

The application, model choices, and system architecture must be approved by the instructor.

Application Goal

Your application must solve a meaningful real-world problem for a clearly identified user or community.

Possible application areas include:

• Assistive technology for blind or low-vision users

• Senior assistance

• Mobility assistance

• Health and wellness

• Sports and athletic performance

• Education

• Animal monitoring or assistance

• Environmental monitoring

• Safety

• Navigation

• Smart environments

• Accessibility

• Human-computer interaction

Other application areas are possible with instructor approval.

IMPORTANT    

Your application must do more than simply recognize an object.

For example:

❌ Not enough: Take a picture → identify "chair."

✅ Better: Detect obstacles → determine which obstacles interfere with the user's path → provide appropriate navigation information.

❌ Not enough: Recognize food.

✅ Better: Identify food → analyze relevant visual properties → reason about the result → provide information useful for the application's intended purpose.

 

 

Mobile Computer Vision Requirement

 

This is a MOBILE COMPUTER VISION PROJECT.

Your final system must contain a functional mobile application that uses the camera as a primary source of visual information from the real world.

The application must acquire: Live camera images/video OR Images captured by the user through the application

A collection of images processed only inside a Colab/Jupyter notebook does NOT satisfy this requirement.

The mobile application must allow a user to:

1. Acquire visual information from the camera.
2. Initiate or control the application's computer vision task.
3. Receive results from the AI/computer vision system.
4. Interact with or act upon those results.

On-Device vs. Cloud Processing   

Not every AI model can or must be execute directly on the mobile device.

Your mobile application may communicate with:

• Cloud AI services
• APIs
• GPU servers
• Backend services
• Databases
• Other appropriate external tools

However, your documentation must clearly identify where models and services are run

 

 

 

 

REQUIRED: Multi-Model AI Architecture

Your application must use multiple AI/computer vision models working together as ONE integrated system.

Minimum Requirement: THREE Model Components

Every project must contain at least:

   1. MODEL A - Student-Selected Trained or Fine-Tuned Vision Model

   2. MODEL B - Fine-Tuned Gemini Multimodal Model

   3. MODEL C - Additional Pretrained Vision / Foundation Model

 

You may use additional models (Model D, Model E, etc.) as needed by the application.

 

 

IMPORTANT - Each model must have a different and clearly defined responsibility. Adding a model simply to satisfy the three-model requirement does not count. For every model, your group must be able to answer: Why does our application need this model? What capability or information does this model contribute that another model does not?

MODEL C DOES NOT NEED TO BE FINE-TUNED.

The point of Model C is to demonstrate that you know how to select and integrate existing foundation models and pretrained vision capabilities into a larger system.

Possible models are many and include:

• OWL-ViT
• Grounding DINO
• SAM / SAM 2
• CLIP
• SigLIP
• DINOv2
• Depth-estimation models
• Pose-estimation models
• OCR models
• Pretrained object detectors
• Pretrained segmentation models
• Other VLMs - Generative models
• Other appropriate vision foundation models

 

**** PARTIAL EXAMPLE ****

Computer Vision / Image Processing

Your application must demonstrate meaningful computer vision processing.

Depending on your application, this may include:

• Image enhancement
• Resizing
• Cropping
• ROI extraction
• Color-space transformations
• Geometric transformations
• Tracking
• Temporal/video processing
• Object detection
• Open-vocabulary detection
• Segmentation
• Depth estimation
• Pose estimation
• OCR
• Visual embeddings
• Image similarity
• Other appropriate techniques

Do not add unnecessary image processing simply to make your pipeline longer.

Every processing step should have a reason for existing.

 

 

 

Data Requirements

Your project must contain appropriate data for developing and evaluating the system.

For each model that you train or fine-tune, you must clearly identify:

TRAINING DATA

VALIDATION DATA

TEST DATA

You must document:

• Where the data came from
• How the data was collected
• Number of examples
• Data format
• Labels/annotations
• Preprocessing
• Training/validation/test split
• Limitations
• Potential biases

You may use public datasets when appropriate.

However, your group should contribute application-specific data unless otherwise approved.

IMPORTANT

Your final test examples must NOT be the same examples used for training or fine-tuning.

 

Mobile User Interface

 

Your mobile application must contain a functional user interface appropriate for its intended users.

At minimum, the interface should allow the user to:

1. Start/control the visual task.
2. Understand when processing is occurring.
3. Receive the system's result.
4. Recover gracefully when the system cannot determine an answer.

Output may include:

• Text
• Graphics
• Bounding boxes
• Segmentation masks
• Audio/speech
• Vibration/haptic feedback
• Alerts
• Other appropriate interaction mechanisms

If your application is designed for an accessibility-related purpose, the interface must be appropriate for the intended users.

 

System Architecture Diagram (Documentation)

 

Your proposal AND final documentation must contain a system architecture diagram.

Your diagram must identify:

📷 Camera / Input

↓

⚙️ Preprocessing

↓

🧠 Each AI/Model Component

↓

🔄 Information Passed Between Components

↓

☁️ Cloud / Backend Services

↓

⚙️ Application Logic

↓

📱 User Output / Action

For EVERY model, document:

 

Requirement	Description
Model	Exact model/architecture
Purpose	Why it exists in your application
Input	What is sent to the model
Output	What the model produces
Location	Mobile device or cloud/backend
Customization	Trained, fine-tuned, or pretrained
Next Step	Where the model's output goes

Project Proposal

Before beginning full implementation, your group must submit a technical proposal.

Your proposal must contain the following sections.

1 Problem and Goal What problem are you solving? Who will use the application? Why is computer vision appropriate? 2 Use Scenario Describe how a real person would use the application. 3 Input and Output What will the camera observe? What information/action will the application provide? 4 System Architecture -GIVE SYSTEM DIAGRAM Provide your proposed multi-model architecture diagram. 5 Model A What model will you train/fine-tune? What task will it perform? Why did you choose it? What data will you use? 6 Model B — fine-tuned Gemini What will Gemini do? Why is this task appropriate for Gemini? What will your fine-tuning examples look like?   7 Model C+ What additional pretrained model(s) will you use? Why? 8 Data Describe your proposed datasets, data collection, annotations, and testing data. 9 Mobile Interface Provide interface mockups. 10 Evaluation Plan Explain how you will evaluate all fine-tuned models and complete mobile application.   11 References Provide references supporting your proposed technical approach.

 

 

 

MODELs Evaluation (Documentation)

You must quantitatively evaluate all your fine-tuned Models (A+B)

Use metrics appropriate for your particular task, such as:

Accuracy • Precision • Recall • F1 • IoU • mAP • Confusion Matrix

or another appropriate metric.

When appropriate, show:

• Training loss
• Validation loss
• Training accuracy
• Validation accuracy
• Learning curves
• Other appropriate measures

You must show both:

✅ Successful Examples

and

❌ Failure Examples

You must discuss why you believe the failures occurred.

SPECIAL NOTE: If the model is a fine-tuned model from an existing public pre-trained model, you must also show the testing results on the pre-trained model for comparison.

 

Full-System Evaluation (Documentation)

Individual models performing well does NOT necessarily mean the application performs well.

You must therefore evaluate the complete integrated mobile vision system.

Your evaluation should address:

• Does the application accomplish its proposed task?
• Does the multi-model architecture work as intended?
• Where do errors propagate between models?
• Which component causes the largest number of failures?
• What is the application's response time?
• Is the application usable in realistic conditions?
• What happens when one of the models produces an incorrect result?

Real-World Camera Testing (Documentation)

Your final application must be tested using real camera input.

Testing should include multiple conditions appropriate for your application.

For example:

Indoor / Outdoor

Bright / Low Light

Different Backgrounds

Different Viewing Angles

Different Distances

Partial Occlusion

Multiple Objects

Motion

Clutter

Previously Unseen Environments

 

 

Not every project needs every condition. Select conditions relevant to your application.You MUST show failures.  

A project demonstration containing only carefully selected successful examples is NOT sufficient evaluation.

 

 

GitHub + Weekly Development

 

Your group must use GitHub for source control and project management.

You must maintain:

• Source-code repository/repositories
• GitHub Issues
• Project tasks
• Meaningful commit history
• Technical documentation

Weekly progress must demonstrate actual technical development.

Simply reporting what you intend to do next week is not sufficient.

Each group member must have identifiable technical contributions to the project.

 

Final Deliverables

 

Your final submission must include all of the following.

1. Mobile Application Source Code --> Turn in URL to GitHub repository 1

Complete source code necessary to build and run the application.

2. Model Development Code --> Turn in URL to GitHub repository 2

All notebooks/scripts/configurations used for:

• Model A training/fine-tuning
• Gemini fine-tuning
• Model evaluation
• Data preparation

3. Data Documentation --> Part of "Documentation Report PDF" uploaded to Canvas (and linked on ReadMe of GitHub repository 1)

Document all datasets used in the project.

4. Model Results --> Part of "Documentation Report PDF" uploaded to Canvas (and linked on ReadMe of GitHub repository 1)

Provide:

• Training results
• Validation results
• Test results
• Metrics
• Graphs
• Successful examples
• Failure examples
• Base Gemini vs. Fine-Tuned Gemini comparison

5. System Documentation --> Part of "Documentation Report PDF" uploaded to Canvas (and linked on ReadMe of GitHub repository 1)

Documentation must contain:

Project Overview

System Architecture

Installation / Setup

Data

Model A Specs

Model B Gemini Fine-Tuning Specs

Additional Models Specs

Mobile Application GUI + specs

Cloud / Backend Specs

Testing/Eval

Results Discussions including Outcomes, Failures &Limitations

Instructions for Reproducing the System

Demonstration Video (uploaded to YouTube and Narrated)

• Your video must demonstrate the ACTUAL MOBILE APPLICATION USING CAMERA INPUT.

• The demonstration must include previously unseen real-world examples.

• Do not show your Colab notebooks or code

7. Final Presentation

Your presentation should tell the story of your project:

Problem → Users → Architecture → Models → Data → Training/Fine-Tuning → Mobile Application → Evaluation → Results → Failures → Lessons Learned