Exercise Learn Vision Image TransformerBy Example
...with Keras using TesnorBoard --- plus Finding/Exploring solutions

PART 1 Learn Training By Example GROUP work    

• Pre-requistes: experience you gained in Lab (Vision Transformer Training) that performs Video Activity Classification

• Task:

  1. FIX (Step 6) the code so that the tensorboard and checkpoint callbacks work and fix the plotting (all in green)

  2. FIX (Step 9) the code to see if there is a way to get this model to export to tflite. In the comments there is a link to another colab ( and some documentation from google) that is geared towards the use of MoViNet for mobile/edge that does support conversion (and they supply some tools for this). I want you to see if you can use this or other resources online to fix our current colab to export to TFlite

     how current colab loads model   # Import the MoViNet model from TensorFlow Models (tf-models-official) for the MoViNet model from official.projects.movinet.modeling import movinet from official.projects.movinet.modeling import movinet_model     #later it creates model (and not shown loads weights from a URL # Set num_classes=600 to load the pre-trained weights from the original model model = movinet_model.MovinetClassifier(backbone=backbone, num_classes=600) model.build([None, None, None, None, 3])

     What the secondary colab does Load a MoViNet KerasLayer from tfhub.dev. def load_movinet_from_hub(model_id, model_mode, hub_version=3):   """Loads a MoViNet model from TF Hub."""   hub_url = f'https://tfhub.dev/tensorflow/movinet/{model_id}/{model_mode}/kinetics-600/classification/{hub_version}'     encoder = hub.KerasLayer(hub_url, trainable=True)     inputs = tf.keras.layers.Input(       shape=[None, None, None, 3],       dtype=tf.float32)

Part 1 Deliverables:

1) turn in the PDF document titled "Part1.pdf" which contains the following

Section 1= Fix for callbacks to work

Section 2 = Fix plot showing in Confusion Matrix

Section 3 = Fix for TFlite export

2) Link to colab that is publically accessible.

Part 2: Find/Explore GROUP work    

• Each group will be assigned ONE of "Image Classification/ Object Detection/ Image Captioning / Video Captioning" Your group will find your favorite Colab using tensorflow and keras that applies a Vision Transformer to the problem.

  • example "tensorflow keras colab Object Detection with Transformers: DETR" colab found and github of this model found from papers with code.
  • NOTE: for Image Classification I want you to modify any colab you find so that it works with the Flower dataset we have from our beginning Colab work on Classification
  • NOTE: for Object Detection I want you to modify any colab you find so that it works with the Cat/Dog dataset you created for our beginning Colab work on Detection.
    
    
    

• You will need to have run the Colab and fix any errors. Make sure the Colab will train (retrain ideally) and make predictions on test data. Add the code to have TensorBoard and Checkpoint callbacks and code to save the model as .keras model and export to TFlite. ALSO make sure you add comments to each "major section" --use titles like "STEP X: blah blah" and give markup text about what is going on in that "section" of colab cells. Make sure you save the output from running the cells.

Part 2 Deliverables:

1) turn in link to colab.

2) turn in the PDF document titled "Part2.pdf" which contains: a discussion of each major STEP in the colab and shows the cell outputs for that sections. The PDF document should have section titles like "STEP X: blah blah" to mimic the markup cell text in the colab. Discuss any problems that you encountered.

3) Present in class

Deliverables

part 1 =Part 1 Deliverables:

1) turn in the PDF document titled "Part1.pdf" which contains the following

Section 1= Fix for callbacks to work

Section 2 = Fix plot showing in Confusion Matrix

Section 3 = Fix for TFlite export

2) Link to colab that is publically accessible.

part 2 =Part 2 Deliverables:

1) turn in link to colab.

2) turn in the PDF document titled "Part2.pdf" which contains: a discussion of each major STEP in the colab and shows the cell outputs for that sections. The PDF document should have section titles like "STEP X: blah blah" to mimic the markup cell text in the colab.

3) Present in class