LSTM Example (w/ webcam)

 

 

****** final notbook that does webcam capture + processing of frames -> prediction + storage *******

 

• ITEM 1: Training with evaluation& run testing daya& Tensorboard
  
  

  • Name of notebook should be LSTM_XX_Training.ipynb

  ITEM 2: Feature Extraction Jupyter Notebook

  • you really have this code but, you are putting it in a separate notebook. This notebook should open up a directory of videos and process each of them with your Feature Extractor CNN (minus softmax layer) and output JSON files of the same name as each video in the same diretory. It is a nice notebook to have seperate so it is easy to bring up and quickly use when preparing data. This NEW jupyter notebook called FeatureExtrationNotebook.ipynb

    Deliverables item 2: video_feature_extraction(youtube), doc_video_feature_extraction.pdf (screenshotes and text explaining them), FeatureExtrationNotebook.ipynb, data_featureExtractoin.zip (zip of directory where did feature extraction, should have videos AND the feature extraction files)

  ITEM 3: Seperate Jupyter Notebook -that will perform batch predections on a directory of videos

  • this notebook has a lot of the same code as item 1 but, does not retrain, it simply loads the new testing files existing in a directory for processing (NOTE: this can be after you have processed them via ITEM 2 separately and hence be the npy files) and then does prediction on them. this NEW jupyter notebook called BatchPredictionNotebook.ipyn.
    
    

ITEM 4: LIVE Testing w/ live Video Capture



• this notebook copies the idea I have shown in this video with this notebook (and corresponding python). It will capture Live Video and process the frames appropriately through the feature extractor and into your LSTM. BUT, you must change it significantly so that it works with YOUR LSTM model instead of the CNN Cat Dog example. See LiveFeatureVector_Collection concept below. this NEW jupyter notebook is called LivePredictionNotebook.ipyn.
  
  

• LiveFeatureVector_Collection Concept:

  • unlike my CNN example, you will have to collect 40 (or whatever sequence number you trained with) feature vectors from the last 40 sampled images in the live Video. You will process each video through your Feature Extractor (CNN minus softmax layer) and then store the vector in a data structure like a Queue (I will call it FV_Queue for "feature vector queue"). Each time a new image comes in you see if it is the Kth (remember you dont take every frame --like in training) image and if so get its Feature Vector and remove the oldest feature vector in queue and add this new Feature Vector to it. Then you package the current set of Feature Vectors in FV_Queue into a tensor input acceptable by your LSTM and call prediction.

    • Storing Results: you must think about how you will store your results. I suggest that you make a unique directory called label_live_timestamp where you store the 40 images used in your 40 feature vector set --call the images img1.jpg to img40.jpg. Create a file called FeatureVectors.txt that is a dump of the 40 feature vectors. Create a file in this directory called results.txt that give the label(your decision), the prediction vector.

      This means that for every set of 40 image frames you send to predictor you will have a separate directory label_live_timestamp (unique because of timestamp). label=class label that the system decided was the label for that 40 imge frames. Example, think about the classes cross_left and cross_right and straight. That means you will have directories called "cross_left_live_20191120-173207" and so on