Reading MV- Chapter 1

• Course Overview,    Introduction lecture
• Unix Tips
• Group Collaboration (work is individual unless specifically denoted as group work)
  

2 Creating Images

• Image formation lecture 1 [pinhole cameras and lenses] and video , Image formation lecture 2 [sampling and quantization] and video

3 Our Visual System + Color

• Human Abilities lecture     video  
• Illumination/ Light. CVA- 2.6,Color Space lecture [basics ]

4 Image/Video Formats

Reading - OPTIONAL Video: Chapter 2 of "Practical Image Processing in C" by C. Lindley, OPTIONAL Image Formats: p.g. 185-187,214-240 of "Practical Image Processing in C" by C. Lindley

• image formats      [image header example ]
• video standards
• display types

Begining Android LAB:A.1-1

5 Simple Image Operations

Reading - MV- Chapter 2

• Point-based Processing Lecture and video    [Point Operators]
• Histogram
• Binary Images

6 Software

• Arrays and Indexing (or see specific data structures/classes used in any package your plan to employ)

LAB 1: Learn more about OpenCV using these Google Colabs. INDIVIDAUL WORK start of class Sep 1

• REQUIRED - must show when called upon in class your Colab up and running:
  • THROUGH STEP 3: Google Colab (not mine): Python + OpenCV calls ( and you can also download the corresponding python notebook)
• OPTIONAL -another example
  • Google Colab (not mine): Simple example on Python and a few OpenCV calls

7 Probability+

Reading -CV-chapter 7, probability (local copy)

• probability basics and video, distribution basics and video, [more basics]
• more yet: MSE and crosscorrelation

8 Histograms, Equalizaton

Reading - MV- Chapter 4 (thresholding)

• Lecture on Histograms and Equalization and video     [Histograms,   Histogram Equalization]

Exercise 2: Learn more about OpenCV using Google Colabs. INDIVIDAUL WORK (NOT TURNED IN)

• REQUIRED - must show when called upon in class your Colab up and running:
  • continued from before...NOW show STEP 4: Google Colab (not mine): Python + OpenCV calls ( and you can also download the corresponding python notebook)

Project 1 - proposal presentation -due next week

OpenCV & Android
LabA.1-2

9 Geometric Relationships:
Scaling/Sizing, Rotation, Mirror Images

Reading -CV-section 6.1, analytical (local copy)

• simple geometric operations      [basics ]
• math-tranformations: linear algebra lecture, part1, part2

10 Area Processing: edges and filters

Reading -MV- Chapter 3,5, OPTIONAL CVA-section 9.7 (edge), OPTIONAL - CV-chapter 8, linearfilters (local copy),CV-chapter 9, edgedetection (local copy)

• Area Filters lecture     [Neighborhoods,    Area Processing]

Exercise 3: Learn more about OpenCV using Google Colabs. INDIVIDAUL WORK

• REQUIRED - must show when called upon in class your Colab up and running: (not recorded)
  • continued from before...go over STEP 5&6: Google Colab (not mine): Python + OpenCV calls ( and you can also download the corresponding python notebook)
  • Add a new cell for a STEP 7 that performs Sobel Edge Detection and displays the output image
  • Add another new cell for a STEP 8 that performs Laplacian of Gaussian Edge Detection (LoG) and displays the output image
  • DUE - turn in to Canvas->Assignments->Exercise 3 (see Canvas for due date&time)
    • the link to your colab AND a screen shot of STEP 7 & 8 running (show both the code AND the displayed output image)

11 Recognition : Deep Learning based techniques

Reading -Deep Learning: MV- Chapter 15

PART 1: CNN(classification) & Object Detection, & Learning/Training your network & RNN/LSTM (video)& Attention to Transformers (part 1 video & part 2)

PART 2: VLMs (foundational) & Using VLMs & OVD & AI workflows n8n intro (n8n AI example & custom models example) & SaaS & more

PART 3: Vision Agentic Computing (as time allows): Google Code Lab (a nutritional Assistant)& CVPR Visual Agents 25 blog

• Agent Visual Assistant codelab (advanced / optional )

LABS:

1) Classification Colab Overview video Part 1 AND video Part 2

2) New Detection Colab (Ultralytics + Roboflow) - if necesary use your own dataset

• (older - Detection Colab and Overview video) -- NOTE this is using Google's AI Edge ModelMaker API which focuses on the creation of mobile ready model (can be converted to TFLite/LiteRT)[Optional, Old] TF2 Object Detection Training and their LONGish blog ---this is not using Google AI Edge
• [Again not using Google AI Edge] TF Object Detection Retraining of ResNet

3) Regression Colab

• [Optional] Beginning deep learning with tensorflow colab (NN + CNN) [ PART1-regression from features w/NN, PART2- classification w/CNN] (and local jupyter notebook version -w/ errors fixed) The code above in the "part 1 Boston Housing Price Prediction" AND "part2" has "errors" because the Tensorflow API changed since its creation. The first error is regarding the movement of tf.train.AdamOptimzer to tf.optimzers.Adam and the solution (search on error to find) shows how you have to modify the code for the newer versions of tensorflow). So, if you find code or get code samples you may be doing this kind of discovery and change. The 2nd error has to do with the keys to the hist[] output in the same part1 --lets see if you can fix that. (hint: print out entire array and see the keys). There will be other places in the colab it tries to access elements with wrong key name.
  

4) Multi Input Multi Output Regression Colab and Overview video

5) Tensorflow implementation of ViT (not training just network coding)

6) Vision Transformer Retraining Colab uses MoViNet for retraining for video activity recognition

7) Fine-tuning Qwen Colab

8) Fine-tuning Gemini: Google Cloud Setup  AND Colab AND Gemini w/ structured concise output tips

9) Simple - Proof of Concept Developer Gemin iAndroid App for base + fine-tuned Gemini call (input text output text ---will need to modify model and input for multi-modal)

Useful Machine Learning Stuff/Tools

• Kaggle (datasets, competitions, resources -own colab like hosting) -Example (past) competition: Google Visual Relationships (bounding box +) Competition
• LabelImg (open source tool for labeling images in PASCAL VOC XML format) , VOTT , from microsoft). CVAT (from Intel) & CVAT Online , labelme (online tool from MIT) and local download labelme tool
  • About use of polygon rather than bounding boxes --> segmentation learning tasks
  • There are other commercial tools (e.g. from Roboflow) , supervise.ly (ai assisted)
  • CVAT allows for you to not only create labels but, to set attributes for them read blog on how to use and take a look at this
• Netron (visualizer ...tensorboard has some)
• Anaconda --easy-to-install package manager, environment manager, and Python distribution with a collection of 1,500+ open source packages with free community support. Anaconda is platform-agnostic, so you can use it whether you are on Windows, macOS, or Linux
• Python install link
• Jupyter Notebook
• JupyterLab is a web-based dev environment for Jupyter notebooks+ and you can optionally install this if you wish    and Video on JupyterLab w/Github
• JupyterLab debugging extension --how you can debug your notebooks.****NEW*****
• SERVICES for labeling (not free): Google labeling service, RoboFlow Pro assistive tool, more --search on your own
• Learning Python resources
  • Python @ w3schools,
  • one of many interactive online consoles (to practice your learning python) ,
  • go to safari index at library.csueastbay.edu and find a book like Learning Python by Mark Lutz

• Training Data Example Labeling
• Python for Training and Jupyter Notebook setup
• Data Augmentation Example (when you have "sparse"data)
  
  

EXERCISES - for group exercises only turn in one person in the group, MUST put everyone's name in the text submission

• (individual part 1/ group part 2) Exercise: Classification - Learn Training BY Example - (50 points - points may change to be determined)
  • PART 1- DUE (see Canvas for due date&time) (INDIVIDUAL WORK)
  • PART-2 DUE (see Canvas for due date&time)
    
    
    
• (group) Exercise: Learn OBJECT DETECTION Training BY Example - (50 points - points may change to be determined) exercise: work on this in your project 3 groups
  
  
  • PART1 DUE (see Canvas for due date&time)(30 points)
    
    
  • PART 2 - DUE (see Canvas for due date&time) (30 points)- Turn in 2 Videos.
    
    
• (group)(OPTIONAL) Deploying a model (SavedModel format) to Tensorflow Serving - Learn how to use Tensorflow Serving and deploy the CLASSIFICATION model you created in PART1 (you may need to recreate it and create code in the codelab to save the model to a SavedModel form --see this modified version of the classification colab that does exporting to a saved model- look for the STEP X: Save the model cells ) to a web endpoint, Use the following Colab to deploy and exercise the deployment to create predictions via a Web REST call This Colab runs the endopint in the Colab VM and it will disappear when the Colab shuts off --what do you have to do to deploy permanently (discuss this). Show screenshots of it running.
  • Alternative - Learn how to train and deploy via REST call on AutoML. Show screenshots of it running + deployed
    
    
    
    
• (group) Fine-tuning Gemini Modify your lab colab so that it uses YOUR custom dataset for your Project 2 to fine-tune Gemini. Use the Google Colab credits to do this. Due (see Canvas for due date&time) (30 points)
  
• (group) Fine-tuning Qwen+Exploration of new NAIR environment/service Run Colab but, now in the NAIRR environment. Part of this is to learn how to use this resource. Create a PDF discussing the setup of this resource. Click for some started info. Due (see Canvas for due date&time) (30 points)
  
  
• (group)(OPTIONAL) Retraining a ViT + exploration
  
  
  
  
• (group- OPTIONAL) Exercise Deeper
  • Exercise: Learn by Example: LSTM for Video Activity Recognition: (turn in partial - results of training + tensorboard + do predictions from python) Turn in ideally deployment in Android App OR MINIMALLY find blogs/githubs/resources that show how to run live an LSTM model (with CNN feature extraction for each frame) using LSTM --- +30 extra points if get Android working
    
    
  • Training on CLOUD Exercise: Re deploy training for one of tdhe previous exercises (your same small group picks based on your Project 3 needs --either CNN or RNN based) on Google Cloud by launching a Virtual Machine using ML (50 points - points may change to be determined) You will need to read and follow official directions
    • we will split into 2 groups:
      • Part 1:(Training) Google Cloud Training
      • Part 2:(Run Time Deployment) Android a model (from exercise above) deployed to Cloud and receiving the result from an Android call (after incomming image processed by your canned CNN). I believe you will have to deploy using a Saved Model format (not h5) https://www.tensorflow.org/guide/saved_model. You will "deploy" using Tensorflow Serving .QUESTION (is this video a solution to getting TFlite to convert our LSTM--one group work on this----only for limited ops or for flow-control like RNNs?? OR look at For models with LSTM/RNNs, you can also try the experimental API OpHint to convert). ROADMAP saying LSTM will come 2019
        • Regarding deployment --- you can see me "starting this" for the lstm-- or look at the possibly easier console based method I started --- either way follow official instructions). Turn into Canvas->Assignments->Ex:Cloud . Create documentation helping other students to reproduce what you did as well as showing it working. Note it seems the AI Platform only supports 1.14 Tensorflow not 2.0 at this moment? ALSO -read here about Google Cloud class API from Java/Android
          
        • Android call to Deployed Google AI Platform Model (under construction)
    
    

Android

LabA.1-3

Exercises A1 & A2

12 Vision as Backend Process (cloud) (On own)

• OpenCV deployed with Docker onto Google Cloud (IaaS)
• OpenCV on AWS Lambda with Python deploy script (Paas)
• OpenCV on AWS Lambda with NodeJS (PaaS)
• Google Vision Library in cloud (specific tasks like ocr, object recognition, face recognition, label detection, not as image processing rich as OpenCV)
• Google Cloud Machine Learning

• Lab 1- part 1: Create Google Cloud VM deployed and run your Object Detection Colab for a few hours. How does this experience compare to Colab Pro? Keep track of the time it took to run for 10 epochs.

  Lab 1- part 2: Using NAIR resources upload your Object Detection Colab and Run there, compare for same number of epochs the experience with Google. keep track of the time it took to run for 10 epochs.

13 Recognition (non deep-learning) : numerous techniques

Reading- Model-Based: MV- chapter 12, Statistical Classifiers: MV- chapter 13, 14, Neural Networks: MV - chapter 13, With OPENCV: IP-Chapter 3,4

• Should we learn Prior Deep Learning ML Techniques?
• Intro,Recognition Algorithms - a sampling
  • Nearest Neighbor   intro video and     
    •  openCV implementation,     github implementation of a KNN for color identification and   colab using skilearn (GO HERE for data) --see KNeighborsClassifier
      
      
  • Statistical Classification Gaussian Mixture Models with Notebook;     simple color Image example, and ; see end for example
    
  • Neural Networks lecture and video understanding the math behind backpropogation
  • Support Vector Machines Video: AND The idea and Simple Math Explanation and Colab using Sklearn and hypothetical geometeric features for Maglinent/Benign cancer and Colab SVM explained & at bottom example with features=principal component analysis on aligned cropped small face images and Python code with HOG features + SVM
  • (OPTIONAL/Historical) Bag of Words (based on Text retrieval ideas)
    • paper on topic
  • BoostingColor
  • Optional: Hidden Markov Models
    
    
• Applications lecture
• intro, more (web)
  • graph matching
  • interpretation tree search
    • demo
  • scene labeling
  • neural networks

Optional

• Do the colab on training a NN (local notebook copy)---simply show me that you have run the last cell of the colab (save a screenshot).
  • actually this is a CNN --but, we are going to ignore the Convolution part of it.
  • Note this code uses Cafe which is a framework for Machine Learning and the definition of the neural network used is in the file downloaded from web and you can see involves a 4 layer fully connected set of neurons at the end.
• colab with pre-built Haar Classifiers for facial features using OpenCV (local jupyter notebook version) AND to understand more abou Haar Cascading Classifier and Training your own Cascading Classifiers in OpenCV tutorial
  
  
• colab with GMM: Gaussian Mixture Models (local notebook version) - first part is done with feature vectors (imagine you have processed your images somehow into 2D feature vector) AND 2nd part does modeling of images of numerals -> performs a PCA (principle component analysis) to reduce dimension of input so dont do on ALL pixels values THIS serves as a "kind" of feature extraction -> GMM -> use the GMM to generate new images of numerals (instead of classification)
  • NOTE: this colab uses sklearn which is a simple machine learning framework for Python (has Nearest Neighbor, GMM, SVM, KMeans clustering and more)

14 3D Imaging

Reading- MV- chapter 16 ,

OPTIONAL: CV--chapter 12, (local copy), chapter 13 stereo (local copy), CV- chapter 24 range (local copy), CVA-section 11.3, Depth from motion with OpenCV (non android version)

• stereo vision
• sensing
• RANSAC for image point matching

15 Motion

Reading - Motion: MV- Chapter 20, Tracking-MV - Chapter 22 (surveillance motion & tracking application), optional advanced: CVA-section 10.2-10.3 optical flow

• Motion Part 1 lecture & Part 2 lecture
• Super naive approach ---image differencing --a first step
• Video - naive approach -- using OpenCV to do image difference, dtect contours --> towards motion detection
• video on using RANSAC fitting parameters to a model (any model, a transformation, motion esitmation, etc)
• paper on using RANSAC for camera parameter motion estimation
• The New OpenCV RANSAC algorithm - a blog
• SLAM - Simultaneous Localization and Mapping
• general &differential

16 Segmentation and Fitting

Reading - MV- Chapter 12 , Fitting: CV-chapter 17, fitting (local copy) (chapter 18 for those who are interested...not covered), IP: p.g. 42-44 (previously assigned - contours). See also OpenCV and contours

• segmentation lecture Video     [segmentation notes]

17 Image Databases (on own)

Reading -Digital Libraries: CV-Chapter 25 , diglib (local copy )

• basic concepts

Week 14

18 Image Noise and intro to the frequency domain

Reading - CV-chapter 8, chapter 10, nonlinearfilts (local copy), m(recommend/ not required Fourier Transform: CVA:section 8.6-8.7 )

• frequency lecture
• intro
• fourier
• sample code
• Noise lecture [kinds of noise and reductions ]

19 Binary Image processing (basic on own)

Reading -MV: chapter 8,9

• binary images and morphology lecture
• binary image: geometric feature lecture
• binary image processing, blob detection, labeling and more
  • geometric features

20 Features and Texture

Reading -Texture:MV-chapter 7 , Features and hough: MV- Chapter 10,11 , SIFT - local features (scale and rotation invariant) READ IP- Chapter 3 , OpenCV is ORB and matching with ORB , paper comparing different Local features for recognition in different viewing conditions(image transformations) (PAMI 2005) , OpenCV and SIFT explained. (in python but, same idea)

Class Lecture, Materials

• texture lecture
• feature detection
• hough transform

Exercise 4: More OpenCV using these Google Colabs. GROUP WORK (Not graded but, demoed)

• continue from YOUR exercise 3 and add a new cell 9 that performs your choice of a feature detector (corner points, hough lines, etc) --discuss and show it.

21 Misc.

• Compression introduction lecture &basics
• PyTorch Mobile from past students

Android

A.0 Android Misc.

Android Studio:

• Android Studio - Download

  Android Studio - User Guide

   

Android

A.1 Android

Android *****NOTE: Some of the links will take you to Mobile Prog. website*****

Learning Labs/Experiences  
LAB:A.1 -1 :Simple First App -CodeLab

  [Shortcut option] Create a project (e.g Empty Views Activity - for simply "empty" interface) AND run your project AND Testing and JUnit creation . SHOW running on BOTH Virtual device AND a real Physical Device.

• Alternative UI development (newer. not with XML files) : using Compose for UI
• Example Kotlin/Java APP(with XML UI files)

LabA.1-2: OpenCV + Android and video [Uses OpenCV for image capture] (on emmulator you may need to have cameras set to emulator not webcam --or use physical device as suggested)

Lab A.1-3: Follow Media Pipe + Android Video AND Android+MediaPipe For Classification example (Github hosted code , Note: MediaPipe is part of Google AI Edge) and to create an Android Classification App.

• AND android studio + Media Pipe setup
  

FOR FOLLOWING INFORMATION --REFER TO LATEST ANDROID DOCUMENTATION

Emulator and AVD Manager and Running on a Real Device  

• Android Emulator, AVD Manager and how to creat an AVD
  • AVD Sample Configs (give me your favorite to add)
  • can take a long time to launch emulator ---snapshots?
• How to Debug/Run on a Physical (real) Device
  • Install APK on Physical Device video demo (from previous student ) YouTube

SDK Manager

• Android SDK manager
  
  

Loading Existing APK, Pushing/Pulling Files

• Loading Existing Android Project (what you need and how to do it)
• Loading Existing APK (application)
• Pulling/Pusing File from/to Device in Emulator

A.2 Android and Android-based Image Capture with Tensorflow MediaPipe ML processing

EXPERIENCES - Go through tutorial and run code given to you ---try to understand it -- SEE Experience: A.1-3 (above)

1. Image Classification w/ MediaPipe (using model.tflite EfficientDet model provided in repository)
2. Object Detection w/ MediaPipe (using model.tflite EfficientDet model provided in repository)
3. For Fun ONLY - Image Generation (text to image) w/ provided model

Exercises - even though these are not graded they must be done

• Exercise A1(not recorded): First, look but do NOT do the following java based codelab. Next, look at the vision quickstart sample from Official MLKit GitHub which is taken from the Google Vision API Sample Application page. Next, use this to modify your previous exercise or find online an alternative App that is a Kotlin/Java blend that performs Object Detection using a predefined supplied mode. Consider using the sample you downloaded when you were doing Experience: A.1-3 (Here is an older Java only version --do not use -- base Android App w/ML ). Use your application for part 2 of Exercise: Learn OBJECT DETECTION. listed above. .
  • NOTE: the sample Java code above works for an SSD Mobilenet model. Remember you are looking for a Kotlin/Java hybrid (why- to make the Android part more "modern"). You will need to make modifications on the code to accomodate other models. See the following and an example that has code to support an EfficientDet-D0 model (again Java based).
    
    
    
• Exercise A2 (required, not recorded): THis exercise is to explore the use of Google Vision API
  
  • GOOGLE CLOUD VISION SERVICE- Add to your aExercise A2 pplication the ability to call the Google Cloud Vision labeling service using HTTP request (as shown in lecture and you will have to use Google Cloud Vision documentation), https://cloud.google.com/vision/overview/docs/get-started

A.3 EXTENDED Android and Git

create new github repository and commit from Android Studio