Tensorflow Lite Setup --EXAMPLE & ISSUES from iDrone (follow tensorflow.org site for current)

 

WHY Tensorflow lite over normal Tensorflow?

 

WHY NOT Tensorflow lite over normal Tensorflow?

DEMO CODE = Follow the directions at https://www.tensorflow.org/lite/guide/android to setup the demo code.

• NEED = android ndk installed and setup in Android Studio File->Project Structure->sdk (and see if ndk is setup there)

• READ HERE FOR ANDROID SDK and NDK versions needed - https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite  (on April 2018 it was SDK > 26 and NDK > 14   yet on here https://www.tensorflow.org/mobile/tflite/demo_android it says android version 5 = sdk 22.  Which one???? In sample apk you can download it uses SDK 26.  However, the manifest from the code says minSDKVersion=21. 

• TRIED on 7.0 Asus and the sample compiled and run.

• TRIED on x86 emulator and would NOT  work.  I tried to add support for x86 for emulator using 2 options shown in http://www.omnibuscode.com/board/board_tools/37435  but, did NOT work.

FILE FORMAT= tflite   for models   START OF THIS ****DO NOT COMPLETELY UNDERSTAND ALL THE INPUT PARAMETERS******

Read the section on Freeze Graph at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite  and notice the steps are

STEP 1: freeze the graph (input .pb file and output .pb file)

The user has to input the .pb and the .ckpt files 

Definitions:
GraphDef (.pb) - a protobuf that represents the TensorFlow training and or computation graph. This contains operators, tensors, and variables definitions.

• CheckPoint (.ckpt) - Serialized variables from a TensorFlow graph. Note, this does not contain the graph structure, so alone it cannot typically be interpreted.

• FrozenGraphDef - a subclass of GraphDef that contains no variables. A GraphDef can be converted to a frozen graphdef by taking a checkpoint and a graphdef and converting every variable into a constant with the value looked up in the checkpoint.

• SavedModel - A collection of GraphDef and CheckPoint together with a signature that labels input and output arguments to a model. A GraphDef and Checkpoint can be extracted from a saved model.

• TensorFlow lite model (.tflite) - a serialized flatbuffer, containing TensorFlow lite operators and Tensors for the TensorFlow lite interpreter. This is most analogous to TensorFlow frozen GraphDefs.

STEP 2: convert frozen .pb file to .tflite (a special formatted file that is “better” for mobile quicker access)

Reason to use SDK 8.1 if can on Asus phone:

Supports Android Neural Network API that will speed up operations on the phone for networks by using the phone’s GPU/hardware acceleration

OPTION 1 (modify Tensorflow lite program) ONCE you have Trained your model and converted to .tflite file do the following:

1. Use the demo app as your base code (add the UVCCamera library and OpenCV to it)

2. This demo code has the Camera2BasicFragment that currently tries to make an instance of the ImageClassifierQuantizedMobileNet

classifier = new ImageClassifierQuantizedMobileNet(getActivity());

3. While you can change between this and an Inception model instance (need to alter the Camera2BasicFragment line in #2)  you need to copy your trained network info over --specifically copy the *.tflite AND a *labels.txt file to the assets directory of the app.  The *labels.txt file is the classes (in order of output vector) used in training. Node code is expecting the file for the model to be in mobilenet_quant_v1_224.tflite  and the labels in labels_mobilenet_quant_v1_224.txt
   
   

4. Alter to take OpenCV frames rather than what currently have and give to stereo processing to get depth (if using this for user detection cnn) and then resize as approprite for fixed size CNN.

5. 
   

OPTION 2 (add Tensorflow lite program to your existing image capture app using UVCCamera code and OpenCV) ONCE you have Trained your model and converted to .tflite file do the following:

1. Use your existing app that captures and processes stereo using the UVCCamera library and OpenCV to it

2. NOW integrate parts of the TensorFlow Lite code such as inside Camera2BasicFragment that currently tries to make an instance of the ImageClassifierQuantizedMobileNet

classifier = new ImageClassifierQuantizedMobileNet(getActivity());

YOU will need to add this to your start of your apps MainActivity.

3. Add your trained Network files to assets: 
   While you can change between this and an Inception model instance (need to alter the Camera2BasicFragment line in #2)  you need to copy your trained network info over --specifically copy the *.tflite AND a *labels.txt file to the assets directory of the app.  The *labels.txt file is the classes (in order of output vector) used in training. Node code is expecting the file for the model to be in mobilenet_quant_v1_224.tflite  and the labels in labels_mobilenet_quant_v1_224.txt

4. NOW in the code you get your new image frame in and AFTER any stereo processing is done you need to 

1.  Resize the input (rgb or depth) to input size needed by CNN

2.  Call the classifyFrame method that I copied from the Camera2BasicsFragment class of the TensorFlow Lite demo app,  where bitmap should be CHANGED below and passed as an input to classifyFrame. NOTE the output is very specific and is a formatted String showing most likely top 3 object recognized.  THIS WILL NEED to change. You are going to have to get output as top detected user, ROI bounding box AND label user_left, user_right, user_front, user_back (whatever the object labels are).   Need to figure out the ImageClassifierQuantizedMobileNet and how to adjust it for region detection and not just identification....have not yet looked at this.

private void classifyFrame(Bitmap bitmap) {

 if (classifier == null || getActivity() == null || cameraDevice == null) {

   showToast("Uninitialized Classifier or invalid context.");

   return;

 }

 classifier.getImageSizeY());

 String textToShow = classifier.classifyFrame(bitmap);

 bitmap.recycle();

 showToast(textToShow);

}

NOTE the ImageClassifierQuantizedMobileNet has the following method that processes the frame on CNN using the tflite object and this is what stipulates it is only returning probabilities of each object class in an array and NOT localization.

protected void runInference() {

 tflite.run(imgData, labelProbArray);

}

And tflite is an instance of org.tensorflow.lite.Interpreter

tflite = new Interpreter(loadModelFile(activity));

Special Note on Quantized Frozen model versus Float --NOT SURE if still true

IMPORTANT - tensorflow lite and SSD support....there is someone who is trying to add this to repository for demo --this would include some code that would then not only give probabilities of each class but, DO SO with Region information given = https://github.com/tensorflow/tensorflow/issues/14731 

See https://github.com/andrewharp/tensorflow/tree/master/tensorflow/contrib/lite/java/demo/app/src/main/java/com/example/android/tflitecamerademo  for maybe their added code???