ALERT: topics
from this list may be removed or changed and new topics added
NOTE: IP = "Android OpenCV" book (see syllabus)
MV = "Computer Vision by Davies" book
CV = "Computer
Vision: A modern approach" book
TF = "Hands-On Computer Vision with Tensor Flow 2" book
NOTE: following is recommended (not required) reading
CVA =
"Computer Vision and Applications: A Guide for Students and Practitioners" book
NOTE: CVPR 2011 selected online papers http://www.cvpapers.com/cvpr2011.html (can get more recent CVPR directly on ACM index of our library.csueastbay.edu site)
1 Introduction to Class, Overview of Imaging ApplicationsReading
Class Lecture, Materials
Work: start Project 1 - research tutorial |
|
2 Creating ImagesReading
Class Lecture, Materials |
|
3 Our Visual System + ColorReading
Class Lecture, Materials |
|
4 Image/Video FormatsReading
Class Lecture, Materials |
|
Begining AndroidLecture: Module A1 (see below) |
|
5 Simple Image OperationsReading
Class Lecture, Materials |
|
6 SoftwareClass Lecture, Materials
Exercise 1: Learn more about OpenCV using these Google Colabs. INDIVIDAUL WORK start of class Sep 1 (not recorded)
|
|
7 ProbabilityReading
Class Lecture, Materials |
|
8 Histograms, EqualizatonReading
Class Lecture, Materials Exercise 2: Learn more about OpenCV using Google Colabs. INDIVIDAUL WORK (NOT TURNED IN)
|
|
Project 1 - proposal presentation -week 3More Begining AndroidLecture: Module A2,A3 |
|
OpenCV & Android
READ: IP -chapter 1 (only creating our application)
|
|
10
Geometric Relationships:
|
|
11 Area Processing: edges and filtersReading
Class Lecture, Materials
Exercise 3: Learn more about OpenCV using Google Colabs. INDIVIDAUL WORK
|
|
17 Recognition Part 2 : Deep Learning based techniquesReading
General Deep Learning & Cloud Vision Materials
Tensorflow 2 w/ Keras
Tensorflow Object Detection API
|
If you think you might like to try the alternative path using Model Maker for your Object Detection Exercise Part 2, check out the new colab I made that uses the mask data.Links to an external site. It uses data in the zip fileLinks to an external site. you will need to grab, unzip and upload to YOUR Google Drive (put it inside a Mask/MaskDirect path) You can see it training below. I will continue to work on it after it finishes with 5 epochs to see that it runs the evaluation and testing cells in the colab. If you choose this alternative I am fine with you using the Kotlin Android sample they provide that uses the ObjectDetector class... again see the Exercise statement info. |
- (group) Exercise Deeper: CHOOSE one of the following (with instructors permission) depending on your Project 3: (50 points) DUE Oct. 25 at 11pm -- I WANT MOST TEAMS TO DO one of the first 2 options --only if you convince me will I allow you to do training on cloud. Create Video of your exercise running and MAKE sure to go through the code AND show the results and any output/files/code created and testing results. Turn in to Canvas ->Assignments ->Exercise Deeper. Ideally your video will be on youtube and you are turning in a URL. It is good to make videos like this and you can even "show" potential employers.
- Exercise: Learn by Example: LSTM for Video Activity Recognition: (turn in partial - results of training + tensorboard + do predictions from python) Turn in ideally deployment in Android App OR MINIMALLY find blogs/githubs/resources that show how to run live an LSTM model (with CNN feature extraction for each frame) using LSTM --- +30 extra points if get Android working
- Exercise A3 : THis exercise is to explore the use of Google Vision API (you will have done work on TensorFlow in exercises listed in Recognition module above). You work on this with your project 3 group. Get the code to work and demo it it in a YouTube Video and turn in the url.:
- GOOGLE CLOUD VISION SERVICE- Add to your android application the ability to call the Google Cloud Vision labeling service using HTTP request (as shown in lecture and you will have to use Google Cloud Vision documentation), https://cloud.google.com/vision/overview/docs/get-started
-
VIDEO Demonstartion: you can simply record it running on an emulator from say your mobile device (be careful you can see it) or find one of the many free screen recording services you can search for on the internet. Alternatively you can run on a real android device AND get a screen recording app (I have used Mobius).
- Training on CLOUD Exercise: Re deploy training for one of the previous exercises (your same small group picks based on your Project 3 needs --either CNN or RNN based) on Google Cloud by launching a Virtual Machine using ML (50 points - points may change to be determined) You will need to read and follow official directions
- we will split into 2 groups:
- Part 1:(Training) Google Cloud Training
- Part 2:(Run Time Deployment) Android a model (from exercise above) deployed to Cloud and receiving the result from an Android call (after incomming image processed by your canned CNN). I believe you will have to deploy using a Saved Model format (not h5) https://www.tensorflow.org/guide/saved_model. You will "deploy" using Tensorflow Serving .QUESTION (is this video a solution to getting TFlite to convert our LSTM--one group work on this----only for limited ops or for flow-control like RNNs?? OR look at For models with LSTM/RNNs, you can also try the experimental API OpHint to convert). ROADMAP saying LSTM will come 2019
- Regarding deployment --- you can see me "starting this" for the lstm-- or look at the possibly easier console based method I started --- either way follow official instructions). Turn into Canvas->Assignments->Ex:Cloud . Create documentation helping other students to reproduce what you did as well as showing it working. Note it seems the AI Platform only supports 1.14 Tensorflow not 2.0 at this moment? ALSO -read here about Google Cloud class API from Java/Android
- Android call to Deployed Google AI Platform Model (under construction)
- Regarding deployment --- you can see me "starting this" for the lstm-- or look at the possibly easier console based method I started --- either way follow official instructions). Turn into Canvas->Assignments->Ex:Cloud . Create documentation helping other students to reproduce what you did as well as showing it working. Note it seems the AI Platform only supports 1.14 Tensorflow not 2.0 at this moment? ALSO -read here about Google Cloud class API from Java/Android
- we will split into 2 groups:
- If using Pytorch over Tensorflow (only allowed if functionality not present) - PYTORCH TRAINING COLAB WITH document AND Conversion to Mobile and Applicaiton with Pytorch Mobile detector model.
OpenCV & Android
Lecture: Module A4 (any remaining items)
Exercise: A3
READ: IP -chapter 1 and chapter 2
Recommended : on Safari index on library.csueastbay.edu read chapter 2 of "OpenCV Android Programming By Example", by A. Mohammad - covers histograms and histogram equalization
Discuss Project 3
17 Recognition Part 1 : numerous techniques
Reading
- Model-Based: MV- chapter 12
- OPTIONAL CV-chapter 20, csppose (local
copy)
- Statistical Classifiers: MV- chapter 13, 14
- OPTIONAL CV-21.0 to 21.3, classifiers (local
copy), CVA-section 15.1-15.3 (not good),
- OPTIONAL CV-21.0 to 21.3, classifiers (local
copy), CVA-section 15.1-15.3 (not good),
- Neural Networks: MV - chapter 13, (optional TF- chapter 1,2)
- OPTIONAL CV-21.4, classifiers (local
copy)
- OPTIONAL CV-21.4, classifiers (local
copy)
- With OPENCV: IP-Chapter 3,4
Class Lecture, Materials
- Intro
- Recognition Algorithms - a sampling
- Nearest Neighbor intro video and
- Statistical Classification Gaussian Mixture Models with Notebook; simple color Image example, and ; see end for example
- Neural Networks lecture and video understanding the math behind backpropogation
- Support Vector Machines Video: AND The idea and Simple Math Explanation and Colab using Sklearn and hypothetical geometeric features for Maglinent/Benign cancer and Colab SVM explained & at bottom example with features=principal component analysis on aligned cropped small face images and Python code with HOG features + SVM
- Bag of Words (based on Text retrieval ideas)
- BoostingColor
- Optional: Hidden Markov Models
- Applications lecture
- intro, more (web)
Optional
- Do the colab on training a NN (local notebook copy)---simply show me that you have run the last cell of the colab (save a screenshot).
- actually this is a CNN --but, we are going to ignore the Convolution part of it.
- Note this code uses Cafe which is a framework for Machine Learning and the definition of the neural network used is in the file downloaded from web and you can see involves a 4 layer fully connected set of neurons at the end.
- colab with pre-built Haar Classifiers for facial features using OpenCV (local jupyter notebook version) AND to understand more abou Haar Cascading Classifier and Training your own Cascading Classifiers in OpenCV tutorial
- colab with GMM: Gaussian Mixture Models (local notebook version) - first part is done with feature vectors (imagine you have processed your images somehow into 2D feature vector) AND 2nd part does modeling of images of numerals -> performs a PCA (principle component analysis) to reduce dimension of input so dont do on ALL pixels values THIS serves as a "kind" of feature extraction -> GMM -> use the GMM to generate new images of numerals (instead of classification)
- NOTE: this colab uses sklearn which is a simple machine learning framework for Python (has Nearest Neighbor, GMM, SVM, KMeans clustering and more)
12 Image Noise and intro to the frequency domain
Reading
- CV-chapter 8, chapter 10, nonlinearfilts (local copy),
- (recommend/ not required Fourier Transform: CVA:section 8.6-8.7 )
Class Lecture, Materials
- frequency:
- Noise lecture
- kinds of noise and reductions
- exercise (not turned in)
13 Binary Image processing
Reading
- MV: chapter 8,9
Class Lecture, Materials
14 Features and Texture
Reading
- Texture:MV-chapter 7
- OPTIONAL CV-chapter 11, texture (local copy),
- Features and hough: MV- Chapter 10,11
- OPTIONAL: CV- chapter 17(local copy) ,
- SIFT - local features (scale and rotation invariant) READ IP- Chapter 3
- ALTERNATIVE that comes part of standard OpenCV is ORB and matching with ORB
- paper comparing different Local features for recognition in different viewing conditions(image transformations) (PAMI 2005)
- OpenCV and SIFT explained. (in python but, same idea)
Class Lecture, Materials
- texture lecture
- feature detection
- features part 2 (attributes of features --see Davies book, 6.10 -shape description by moments, )
- hough transform
- exercise (not turned in)
Exercise 4: More OpenCV using these Google Colabs. GROUP WORK (Not graded but, demoed)
- continue from YOUR exercise 3 and add a new cell 9 that performs your choice of a feature detector (corner points, hough lines, etc) --discuss and show it.
Features: OpenCV & Android -features
Reading: IP - Chapter 2, Chapter 3: SIFT & more , Chapter 4 - Cascade filters(HOG), Chapter Chapter 7 :OCR with KNN, OCR with Support Vector Machines (SVM)
Recommended : on Safari index on library.csueastbay.edu read chapter 3 & 6 of "OpenCV Android Programming By Example", by A. Mohammad - covers edges, shapes with Hough transform,corner detection, cascade classifier ( using Haar-like features and adaptive boosting)
FYI: there is an OCRTesseract class in OpenCV that does OCR relatively well that I recommend using if OCR is not a main topic you want to alter do research on. This chapter's reading is more about how to do KNN and SVM
24 Vision as Backend Process
Cloud and Vision
- OpenCV deployed with Docker onto Google Cloud (IaaS)
- For PaaS that don't include OpenCV support directly
- OpenCV Java bindings packaged with native libraries, delivered as a turn-key Maven dependency -- good if instead of building a project for each platform you provide the OpenCV binaries for inclusion as a dependency in Maven projects.
- OpenCV on AWS Lambda with Python deploy script (Paas)
- OpenCV on AWS Lambda with NodeJS (PaaS)
- Google Vision Library in cloud (specific tasks like ocr, object recognition, face recognition, label detection, not as image processing rich as OpenCV)
- Google Cloud Machine Learning
-
LAB (done in class potentially finish at home): Show the VM deployed and run EITHER YOUR PROJECT 3 colab OR SOME PAST Tensorflow training based Colab for a few hours and show it results of training --email instructor by 11pm on 11/18 EACH person. NOTE: only one person in your group should be training the Project 3 Colab --so decide this amongst your group. Others will choose a past exercise colab to use.
>>> turn in to Canvas ->Assignments -> Colab Running on Google Cloud VM
1) screen shot of training after say 1 hour running with the GCE VM (Google Compute Engine VM), (show me the training cell results)
2) screenshot of colab where hover mouse over connection pulldown showing you are connected to a GCE VM
3) screen shot showing the VM in the Google Compute Engine VM conole window.
16 Segmentation and Fitting
Reading
- MV- Chapter 12
- OPTIONAL CV-chapter 16, clustering (local copy)
- Fitting: CV-chapter 17, fitting (local copy) (chapter 18 for those who are interested...not covered)
- (recommended / not required p.g. 113-125, "A Guided Tour of Computer Vision" by Nalwa. )
- IP: p.g. 42-44 (previously assigned - contours). See also OpenCV and contours
Class Lecture, Materials
18 Data Structures in Vision
Reading - none
Class Lecture, Materials
19 Motion
Reading
- Motion: MV- Chapter 20,
- Tracking-MV - Chapter 22 (surveillance motion & tracking application)
Class Lecture, Materials
- Motion Part 1 lecture
- Part 2 lecture
- Super naive approach ---image differencing --a first step
- Video - naive approach -- using OpenCV to do image difference, dtect contours --> towards motion detection
- video on using RANSAC fitting parameters to a model (any model, a transformation, motion esitmation, etc)
- paper on using RANSAC for camera parameter motion estimation
- The New OpenCV RANSAC algorithm - a blog
- SLAM - Simultaneous Localization and Mapping
- general
- differential
- advanced: CVA-section 10.2-10.3 optical flow
20 3D Imaging
Reading
- MV- chapter 16
- OPTIONAL: CV--chapter 12, (local copy), chapter 13 stereo (local copy), CV- chapter 24 range (local copy), CVA-section 11.3, Depth from motion with OpenCV (non android version)
Class Lecture, Materials
- sensing
- stereo vision
- RANSAC for image point matching
- shape from shading
- reconstruction
21 Image Databases
Reading
- Digital Libraries: CV-Chapter 25 , diglib (local copy )
Class Lecture, Materials
22 Applications in work - face detection & surveillance & towards self-driving cars - case studies from book
Reading
- MV - Chapter 21, 22, 23
Class Lecture, Materials
23 Compression (go over on own)
Reading
Class Lecture, Materials
A.1 Android
Android *****NOTE: Many of the links will take you to Mobile Prog. website*****
- Beginning Android and How to create an Android Project
- LECTURE: Intro
- asPowerPoint alone, as MP4 Video OR as YouTube
- asPowerPoint alone, as MP4 Video OR as YouTube
- LECTURE: START Programming: (HOW to Create a Project in Android Studio)
- as Powerpoint, , as MP4 Video OR as YouTube
- LECTURE: Intro
NEW***Demo - overview of using Android Studio: MP4 Video OR as YouTube
Emulator and AVD Manager and Running on a Real Device
- Android Emulator, AVD Manager and how to creat an AVD
- AVD Sample Configs (give me your favorite to add)
- can take a long time to launch emulator ---snapshots?
- How to Debug/Run on a Physical (real) Device
- Install APK on Physical Device video demo (from previous student ) YouTube
SDK Manager
Loading Existing APK, Pushing/Pulling Files
- Loading Existing Android Project (what you need and how to do it)
- Loading Existing APK (application)
- Pulling/Pusing File from/to Device in Emulator
- Exercise A1:(30 points) Install on your laptop the AndroidStudio IDE and Android SDKs, Create the Hello World code. app. Demonstrate working in BOTH an emulator AND on an Android Physical device DUE -before start of class Sept 8 turn in to Canvas->Assignments->Exercise A1 a screen shot of it running on the emulator(15 points)AND a second screen shot of it running on the physical device (10 points). I do not want the code
Useful Apps / Tools (including important SCREEN RECORDER)
A.2 Android the Interface (Activity, Layout and Views)
Android
- LECTURE: more ideas and review of Android Activity
- as Powerpoint alone, as MP4, as YouTube
- LECTURE: Android Interface
- as Power point, as MP4 , as YouTube
A.3 Android and event handling and intents
Android
- LECTURE: Android more on Event Handling
- as Power point, as MP4 , as YouTube
A.4 Android and Android-based Image Capture with Tensorflow ML processing
WE ARE USING this form of Camera CAPTURE this semester
- LECTURE: Android using the Camera lecture using Android API, as MP4, as YouTube
- LECTURE: Going over the baseMLAndroidAppComptuerVision repository code example (YouTube video overview of code) that uses Tensorflow ML for both Classification and ObjectDetection.
- Camera Issues and Fake Image
- Camera api 2
- READ latest on developer
- Basics Overview
- Building Your Own Camera App
- Newer android.hardware.camera2 (only added API 21) and tutorial on camera2 api
- Web pages:
- Adding OpenCV as a Module to your existing Android Application >>>>>>>FIX to current Android Studio Problem
A.4.2 Android and OpenCV centric frame capture featuring OpenCV calls
OpenCV API for version 4.0.1 (if you are using a different version search for api online)
3rd party OpenCV videos: (some desktop not android examples) -- showing the power of OpenCV
- object tracking
- hand recognition
- face recognition
- face detection
- image processing and another image processing
- SIFT operator
- OCR
- Sodoku Solver
- Object Recognition
- with a "robot"
- for the blind: color detector reading signs money reading
crowd sourcing + "be my eyes" everything - many more ---search "OpenCV Android XXX"
Non-free OpenCV extesions/modules
Some of the OpenCV modules are no longer free in later versions of OpenCV
- Incorporating NonFree Module (opencv3* has non-open/free modules , some free for research)
- word document showing steps for OpenCV3.*
- NOTE: ALTERNATIVE for SIFT/SURF (not free) that comes part of standard OpenCV is ORB and matching with ORB
Camera Capture in Android using OpenCV: WE are NOT using this form of Camera CAPTURE this semester...
VIDEOS:
1) Discussion of OpenCV w/ Android setup and code MP4 YouTube
2) Demo OpenCVAndroid proj. Create from Scratch:MP4 YouTube
3) Demo - overview of using Android Studio: MP4 YouTube
Coding Basics with OpenCV camera capture
OTHER
- Tips
-
Writing your Own pixel manipulation algorithms and Speed issues
-
Special Note on running on Device
- Setup for using Tesseract (OCR) with OpenCV
- Samples Downloaded with OpenCV:
-
optional: consider running Sample – image-manipulations – this example demonstrates how OpenCV can be used as an image processing and manipulation library.
Exercises
- Exercise A2(30 points): Using the base Android App w/ML example, modify it so it adds another button on the top (yes not the nicest GUI design) that launches a new activity called ImageProcessActivity that takes every image captured by Android camera and runs it through OpenCV's edge detector of your choice and simply displays the results.NOW integrated into part 2 of Exercise: Learn OBJECT DETECTION. No longer a separate exercise. . Get the code to work and demo it it in a YouTube Video and upload this URL to Canvas ->Assignments>A3 The Video MUST be clearly visbile and audio good quality.
This exercise will:- demonstrate your familiarity with basic Android code.
- make your familiar specifically with the code in the base Android App w/ML and give you practice running it on a real device (and/or in simulator)
- demonstrate your ability to include OpenCV into an Android application and make a call to it
- NOTE: you can also look to the offical Tensorflow Android example website
- NOTE: the code above works for an SSD Mobilenet model. You will need to make modifications on the code to accomodate other models. See the following and an example that has code to support an EfficientDet-D0 model
- Exercise A3 (60 points): THis exercise is to explore the use of Google Vision API (you will have done work on TensorFlow in exercises listed in Recognition module above). You work on this with your project 3 groupNot assigned but a possible option for Exercise Deeper above. Get the code to work and demo it it in a YouTube Video and upload this URL to Canvas->Assignments->A3 WARNING: while you have some material to help you with this exercise part of this exercise is finding and using online documentation to help you. The Video MUST be clearly visbile and audio good quality.
- GOOGLE CLOUD VISION SERVICE- Add to your application in TENSORFLOW B and in same small groups, add to your exercise A2 the ability to call the Google Cloud Vision labeling service using HTTP request (as shown in lecture and you will have to use Google Cloud Vision documentation), https://cloud.google.com/vision/overview/docs/get-started
A.4EXTENDED Android and Git
Demo - Git and Android Studio
- create new github repository and commit from Android Studio
- create new bitbucket repository and commit from Android Studio (if versioning controll not yet on in your project do VCS->Enable Versioning Control prior to what is shown in video)
REST OF MATERIAL WILL NOT BE COVERED BUT HERE FOR YOUR INTEREST
Wavelets
Reading
Class Lecture, Materials
Speech
Class Lecture, Materials
Fuzzy Image Processing
Reading
- CVA-Chapter 16
Camera Calibration
Reading
- CV-section 6.3-6.4, analytical (local copy) ,
- CVA- Chapter 6
HTML basics
Class Lecture, Materials
- URL and Hostnames
- REVIEW: About WWW browsing, HelperApplications
- FreeSpeech, powerpoint
- HTML intro
- Link to CS2020 and CS3520
K.1 Kinect
Kinect
- Kinect for Windows Sensor Overview
- Microsoft Kinect Getting Started (for version 1.8) ( this will reference how to dowload SDK, runtime tools and setup your sensor)
- Visual Studio and Microsoft Visual Studio Online for Code Versioning, Collaboration and Cloud Hosting
- Kinect Developer Site (and SW)
- Exercises:
- Ex K1: Download Kinect Studio and Play --- recorder showing color, depth
- Ex K2: Download Code currently the C# sample called "Skeleton Basics -WPF C#" , see http://www.microsoft.com/en-us/kinectforwindows/develop/sample-code.aspx or if link has changed search ON MICROSOFT page for it. Turn in screen shots of running
- this will perform skeleton tracking and display it. Alter the code to write the Skeleton Tracking ID (you will need to read about Microsoft.Kinect.Skeleton class ---see http://msdn.microsoft.com/en-us/library/dn791989.aspx)