LSTM Variations on Project 3

 

 

****** final notbook that does webcam capture + processing of frames -> prediction + storage *******

 

Motivating Issue:

• unfortunately, the support to convert LSTM models into TFLite did not happen as predicted so, we went to deploying in Cloud and attempting to call the Google deployed ML model via Android without success --we were able to process Google credentials (OAuth) and make a request to get info about model, but, the packaging of the data to the model was not successful. After many, many hours (thank you to the students --Nayan, Thomas, Raymond, Jayesh.... ) there is no alternative but, to go to a different Python based approach.

 

What you must create & turn in:

*** Note you are doing a series of new Jupyter Notebooks including a Live App (Item 4) to replace

• ITEM 1: Training with evaluation& run testing daya& Tensorboard
  
  

  • there is no change here...but, you must include Tensorboard and your video should show different aspects, Name of notebook should be LSTM_XX_Training.ipynb (where XXX is your topic "crosswalk",etc)

    Deliverables item 1: video_training (youtube), doc_training.pdf (screenshots and text showing results), LSTM_XX_Training.ipynb (name of jupyter notebook), data.zip (zip of data directory)

  ITEM 2: Feature Extraction Jupyter Notebook

  • you really have this code but, you are putting it in a separate notebook. This notebook should open up a directory of videos and process each of them with your Feature Extractor CNN (minus softmax layer) and output JSON files of the same name as each video in the same diretory. It is a nice notebook to have seperate so it is easy to bring up and quickly use when preparing data. This NEW jupyter notebook called FeatureExtrationNotebook.ipynb

    Deliverables item 2: video_feature_extraction(youtube), doc_video_feature_extraction.pdf (screenshotes and text explaining them), FeatureExtrationNotebook.ipynb, data_featureExtractoin.zip (zip of directory where did feature extraction, should have videos AND the feature extraction files)

  ITEM 3: Seperate Jupyter Notebook -that will perform batch predections on a directory of videos

  • this notebook has a lot of the same code as item 1 but, does not retrain, it simply loads the new testing files existing in a directory for processing (NOTE: this can be after you have processed them via ITEM 2 separately and hence be the npy files) and then does prediction on them. this NEW jupyter notebook called BatchPredictionNotebook.ipyn.
    
    Deliverables item 3: video_batch_extraction(youtube), doc_video_batch.pdf (screenshotes and text explaining them), newtestdata.zip, BatchPredictionNotebook.ipyn.

    • newtestdata = you will create a separate directory of videos called testing_videos (have 2 videos for each class) that you must create and they should be unique --not used in training/eval/testing of ITEM 1.

 

 

ITEM 4: LIVE Testing w/ live Video Capture



• this notebook copies the idea I have shown in this video with this notebook (and corresponding python). It will capture Live Video and process the frames appropriately through the feature extractor and into your LSTM. BUT, you must change it significantly so that it works with YOUR LSTM model instead of the CNN Cat Dog example. See LiveFeatureVector_Collection concept below. this NEW jupyter notebook is called LivePredictionNotebook.ipyn.
  
  Deliverables item 4: video_live(youtube), doc_video_live.pdf (screenshotes and text explaining them),results.zip (zip up of results directory),LivePredictionNotebook.ipyn

• LiveFeatureVector_Collection Concept:

  • unlike my CNN example, you will have to collect 40 (or whatever sequence number you trained with) feature vectors from the last 40 sampled images in the live Video. You will process each video through your Feature Extractor (CNN minus softmax layer) and then store the vector in a data structure like a Queue (I will call it FV_Queue for "feature vector queue"). Each time a new image comes in you see if it is the Kth (remember you dont take every frame --like in training) image and if so get its Feature Vector and remove the oldest feature vector in queue and add this new Feature Vector to it. Then you package the current set of Feature Vectors in FV_Queue into a tensor input acceptable by your LSTM and call prediction.

    • Storing Results: you must think about how you will store your results. I suggest that you make a unique directory called label_live_timestamp where you store the 40 images used in your 40 feature vector set --call the images img1.jpg to img40.jpg. Create a file called FeatureVectors.txt that is a dump of the 40 feature vectors. Create a file in this directory called results.txt that give the label(your decision), the prediction vector.

      This means that for every set of 40 image frames you send to predictor you will have a separate directory label_live_timestamp (unique because of timestamp). label=class label that the system decided was the label for that 40 imge frames. Example, think about the classes cross_left and cross_right and straight. That means you will have directories called "cross_left_live_20191120-173207" and so on