Intel的OpenVINO的安装与示例运行(Intel CPU GPU MYRIAD)

东方耀

Intel的OpenVINO的安装与示例运行(Intel CPU GPU MYRIAD)

半导体厂商开发的硬件再怎么厉害，也需要软件工具的加持，重复制造轮子不是一个好主意，为了充分挖掘处理器的性能，各个厂家都发布了各种软件框架和工具，比如Intel的OpenVINO，Nvidia的TensorRT等等。

这里重点介绍英特尔发布的针对AI工作负载的一款部署神器--OpenVINO。

我们有了各种开源框架，比如tensorflow，pytorch，mxnet，caffe2等，为什么还要推荐OpenVINO来作为部署工具呢？

OpenVINO是一个Pipeline工具集，同时可以兼容各种开源框架训练好的模型，拥有算法模型上线部署的各种能力，只要掌握了该工具，你可以轻松的将预训练模型在Intel的CPU上快速部署起来


对于AI工作负载来说，OpenVINO提供了深度学习推理套件（DLDT)，该套件可以将各种开源框架训练好的模型进行线上部署，除此之外，还包含了图片处理工具包OpenCV，视频处理工具包Intel Media SDK


在做推理的时候，大多数情况需要前处理和后处理，前处理如通道变换，取均值，归一化，Resize等，后处理是推理后，需要将检测框等特征叠加至原图等，都可以使用OpenVINO工具套件里的API接口完成。

DLDT分为两部分：模型优化器(Model Optimizer)   推理引擎(Inference Engine)


模型优化器是一个python脚本工具，用于将开源框架训练好的模型转化为推理引擎可以识别的中间表达，其实就是两个文件，xml和bin文件，前者是网络结构的描述，后者是权重文件。模型优化器的作用包括压缩模型和加速，比如，去掉推理无用的操作(Dropout)，层的融合(Conv + BN + Relu)，以及内存优化。

使用模型优化器，从主流的开发框架(如Caffe和TensorFlow)中导入训练的模型, 自动剪枝、量化和图层压缩模型, 从而在FPGA上优化执行


了解FPGA是如何用于加速深度神经网络的: https://www.altera.com/solutions ... gence/overview.html




推理引擎是一个支持C\C++和python的一套API接口，需要开发人员自己实现推理过程的开发，开发流程其实非常的简单

这是一款非常给力的专门做推理的工具，并且有intel在不停的开发和优化新的网络结构，有人维护和开发这件事很重要



安装教程：https://docs.openvinotoolkit.org ... openvino_linux.html


pip config set global.index-url https://pypi.douban.com/simple/


run install_prerequisites.sh venv {caffe|tf|mxnet|kaldi|onnx}


OpenVINO™ Toolkit Intel's Pre-Trained Models :
https://docs.openvinotoolkit.org ... ls_intel_index.html



Adding jiang to the video group...

Installation completed successfully.

Next steps:
Add OpenCL users to the video group: 'sudo usermod -a -G video USERNAME'
   e.g. if the user running OpenCL host applications is foo, run: sudo usermod -a -G video foo
   Current user has been already added to the video group

If you use 8th Generation Intel® Core™ processor, you will need to add:
   i915.alpha_support=1
   to the 4.14 kernel command line, in order to enable OpenCL functionality for this platform.


MYRIAD计算棒 只能 FP16   precisions FP16

东方耀

1. ###################################################
   
2. 
   
3. Convert a model with Model Optimizer
   
4. 
   
5. Run python3 /home/jiang/intel/openvino_2020.4.287/deployment_tools/open_model_zoo/tools/downloader/converter.py --mo /home/jiang/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py --name squeezenet1.1 -d /home/jiang/openvino_models/models -o /home/jiang/openvino_models/ir --precisions FP16
   
6. 
   
7. ========== Converting squeezenet1.1 to IR (FP16)
   
8. Conversion command: /usr/bin/python3 -- /home/jiang/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py --framework=caffe --data_type=FP16 --output_dir=/home/jiang/openvino_models/ir/public/squeezenet1.1/FP16 --model_name=squeezenet1.1 '--input_shape=[1,3,227,227]' --input=data '--mean_values=data[104.0,117.0,123.0]' --output=prob --input_model=/home/jiang/openvino_models/models/public/squeezenet1.1/squeezenet1.1.caffemodel --input_proto=/home/jiang/openvino_models/models/public/squeezenet1.1/squeezenet1.1.prototxt
   
9. 
   
10. Model Optimizer arguments:
    
11. Common parameters:
    
12.         - Path to the Input Model:         /home/jiang/openvino_models/models/public/squeezenet1.1/squeezenet1.1.caffemodel
    
13.         - Path for generated IR:         /home/jiang/openvino_models/ir/public/squeezenet1.1/FP16
    
14.         - IR output name:         squeezenet1.1
    
15.         - Log level:         ERROR
    
16.         - Batch:         Not specified, inherited from the model
    
17.         - Input layers:         data
    
18.         - Output layers:         prob
    
19.         - Input shapes:         [1,3,227,227]
    
20.         - Mean values:         data[104.0,117.0,123.0]
    
21.         - Scale values:         Not specified
    
22.         - Scale factor:         Not specified
    
23.         - Precision of IR:         FP16
    
24.         - Enable fusing:         True
    
25.         - Enable grouped convolutions fusing:         True
    
26.         - Move mean values to preprocess section:         False
    
27.         - Reverse input channels:         False
    
28. Caffe specific parameters:
    
29.         - Path to Python Caffe* parser generated from caffe.proto:         /home/jiang/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo/front/caffe/proto
    
30.         - Enable resnet optimization:         True
    
31.         - Path to the Input prototxt:         /home/jiang/openvino_models/models/public/squeezenet1.1/squeezenet1.1.prototxt
    
32.         - Path to CustomLayersMapping.xml:         Default
    
33.         - Path to a mean file:         Not specified
    
34.         - Offsets for a mean file:         Not specified
    
35. Model Optimizer version:        
    
36. [ WARNING ]  
    
37. Detected not satisfied dependencies:
    
38.         protobuf: installed: 3.0.0, required: >= 3.6.1
    
39. 
    
40. Please install required versions of components or use install_prerequisites script
    
41. /home/jiang/intel/openvino_2020.4.287/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_caffe.sh
    
42. Note that install_prerequisites scripts may install additional components.
    
43. 
    
44. [ SUCCESS ] Generated IR version 10 model.
    
45. [ SUCCESS ] XML file: /home/jiang/openvino_models/ir/public/squeezenet1.1/FP16/squeezenet1.1.xml
    
46. [ SUCCESS ] BIN file: /home/jiang/openvino_models/ir/public/squeezenet1.1/FP16/squeezenet1.1.bin
    
47. [ SUCCESS ] Total execution time: 3.27 seconds.
    
48. [ SUCCESS ] Memory consumed: 85 MB.
    
49. 
    
50. 
    
51. 
    
52. ###################################################
    
53. 
    
54. Build Inference Engine samples
    
55. 
    
56. -- The C compiler identification is GNU 7.5.0
    
57. -- The CXX compiler identification is GNU 7.5.0
    
58. -- Check for working C compiler: /usr/bin/cc
    
59. -- Check for working C compiler: /usr/bin/cc -- works
    
60. -- Detecting C compiler ABI info
    
61. -- Detecting C compiler ABI info - done
    
62. -- Detecting C compile features
    
63. -- Detecting C compile features - done
    
64. -- Check for working CXX compiler: /usr/bin/c++
    
65. -- Check for working CXX compiler: /usr/bin/c++ -- works
    
66. -- Detecting CXX compiler ABI info
    
67. -- Detecting CXX compiler ABI info - done
    
68. -- Detecting CXX compile features
    
69. -- Detecting CXX compile features - done
    
70. -- Looking for C++ include unistd.h
    
71. -- Looking for C++ include unistd.h - found
    
72. -- Looking for C++ include stdint.h
    
73. -- Looking for C++ include stdint.h - found
    
74. -- Looking for C++ include sys/types.h
    
75. -- Looking for C++ include sys/types.h - found
    
76. -- Looking for C++ include fnmatch.h
    
77. -- Looking for C++ include fnmatch.h - found
    
78. -- Looking for strtoll
    
79. -- Looking for strtoll - found
    
80. -- Found InferenceEngine: /home/jiang/intel/openvino_2020.4.287/deployment_tools/inference_engine/lib/intel64/libinference_engine.so (Required is at least version "2.1")
    
81. -- Configuring done
    
82. -- Generating done
    
83. -- Build files have been written to: /home/jiang/inference_engine_samples_build
    
84. Scanning dependencies of target format_reader
    
85. Scanning dependencies of target gflags_nothreads_static
    
86. [  9%] Building CXX object thirdparty/gflags/CMakeFiles/gflags_nothreads_static.dir/src/gflags.cc.o
    
87. [ 18%] Building CXX object thirdparty/gflags/CMakeFiles/gflags_nothreads_static.dir/src/gflags_completions.cc.o
    
88. [ 27%] Building CXX object thirdparty/gflags/CMakeFiles/gflags_nothreads_static.dir/src/gflags_reporting.cc.o
    
89. [ 36%] Building CXX object common/format_reader/CMakeFiles/format_reader.dir/MnistUbyte.cpp.o
    
90. [ 45%] Building CXX object common/format_reader/CMakeFiles/format_reader.dir/bmp.cpp.o
    
91. [ 54%] Building CXX object common/format_reader/CMakeFiles/format_reader.dir/format_reader.cpp.o
    
92. [ 63%] Building CXX object common/format_reader/CMakeFiles/format_reader.dir/opencv_wraper.cpp.o
    
93. [ 72%] Linking CXX shared library ../../intel64/Release/lib/libformat_reader.so
    
94. [ 72%] Built target format_reader
    
95. [ 81%] Linking CXX static library ../../intel64/Release/lib/libgflags_nothreads.a
    
96. [ 81%] Built target gflags_nothreads_static
    
97. Scanning dependencies of target classification_sample_async
    
98. [ 90%] Building CXX object classification_sample_async/CMakeFiles/classification_sample_async.dir/main.cpp.o
    
99. [100%] Linking CXX executable ../intel64/Release/classification_sample_async
    
100. [100%] Built target classification_sample_async
     
101. 
     
102. 
     
103. ###################################################
     
104. 
     
105. Run Inference Engine classification sample
     
106. 
     
107. Run ./classification_sample_async -d CPU -i /home/jiang/intel/openvino/deployment_tools/demo/car.png -m /home/jiang/openvino_models/ir/public/squeezenet1.1/FP16/squeezenet1.1.xml
     
108. 
     
109. [ INFO ] InferenceEngine:
     
110.         API version ............ 2.1
     
111.         Build .................. 2020.4.0-359-21e092122f4-releases/2020/4
     
112.         Description ....... API
     
113. [ INFO ] Parsing input parameters
     
114. [ INFO ] Parsing input parameters
     
115. [ INFO ] Files were added: 1
     
116. [ INFO ]     /home/jiang/intel/openvino/deployment_tools/demo/car.png
     
117. [ INFO ] Creating Inference Engine
     
118.         CPU
     
119.         MKLDNNPlugin version ......... 2.1
     
120.         Build ........... 2020.4.0-359-21e092122f4-releases/2020/4
     
121. 
     
122. [ INFO ] Loading network files
     
123. [ INFO ] Preparing input blobs
     
124. [ WARNING ] Image is resized from (787, 259) to (227, 227)
     
125. [ INFO ] Batch size is 1
     
126. [ INFO ] Loading model to the device
     
127. [ INFO ] Create infer request
     
128. [ INFO ] Start inference (10 asynchronous executions)
     
129. [ INFO ] Completed 1 async request execution
     
130. [ INFO ] Completed 2 async request execution
     
131. [ INFO ] Completed 3 async request execution
     
132. [ INFO ] Completed 4 async request execution
     
133. [ INFO ] Completed 5 async request execution
     
134. [ INFO ] Completed 6 async request execution
     
135. [ INFO ] Completed 7 async request execution
     
136. [ INFO ] Completed 8 async request execution
     
137. [ INFO ] Completed 9 async request execution
     
138. [ INFO ] Completed 10 async request execution
     
139. [ INFO ] Processing output blobs
     
140. 
     
141. Top 10 results:
     
142. 
     
143. Image /home/jiang/intel/openvino/deployment_tools/demo/car.png
     
144. 
     
145. classid probability label
     
146. ------- ----------- -----
     
147. 817     0.6853030   sports car, sport car
     
148. 479     0.1835197   car wheel
     
149. 511     0.0917197   convertible
     
150. 436     0.0200694   beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon
     
151. 751     0.0069604   racer, race car, racing car
     
152. 656     0.0044177   minivan
     
153. 717     0.0024739   pickup, pickup truck
     
154. 581     0.0017788   grille, radiator grille
     
155. 468     0.0013083   cab, hack, taxi, taxicab
     
156. 661     0.0007443   Model T
     
157. 
     
158. [ INFO ] Execution successful
     
159. 
     
160. [ INFO ] This sample is an API example, for any performance measurements please use the dedicated benchmark_app tool
     
161. 
     
162. 
     
163. ###################################################
     
164. 
     
165. Demo completed successfully.
     
166. 
     
167. 
     
168. 
     
169. 
     
170. 
     
171. Run Inference Engine security_barrier_camera demo
     
172. 
     
173. Run ./security_barrier_camera_demo -d CPU -d_va CPU -d_lpr CPU -i /home/jiang/intel/openvino/deployment_tools/demo/car_1.bmp -m /home/jiang/openvino_models/ir/intel/vehicle-license-plate-detection-barrier-0106/FP16/vehicle-license-plate-detection-barrier-0106.xml -m_lpr /home/jiang/openvino_models/ir/intel/license-plate-recognition-barrier-0001/FP16/license-plate-recognition-barrier-0001.xml -m_va /home/jiang/openvino_models/ir/intel/vehicle-attributes-recognition-barrier-0039/FP16/vehicle-attributes-recognition-barrier-0039.xml
     
174. 
     
175. [ INFO ] InferenceEngine: 0x7f73d5aef030
     
176. [ INFO ] Files were added: 1
     
177. [ INFO ]     /home/jiang/intel/openvino/deployment_tools/demo/car_1.bmp
     
178. [ INFO ] Loading device CPU
     
179.         CPU
     
180.         MKLDNNPlugin version ......... 2.1
     
181.         Build ........... 2020.4.0-359-21e092122f4-releases/2020/4
     
182. 
     
183. [ INFO ] Loading detection model to the CPU plugin
     
184. [ INFO ] Loading Vehicle Attribs model to the CPU plugin
     
185. [ INFO ] Loading Licence Plate Recognition (LPR) model to the CPU plugin
     
186. [ INFO ] Number of InferRequests: 1 (detection), 3 (classification), 3 (recognition)
     
187. [ INFO ] 4 streams for CPU
     
188. [ INFO ] Display resolution: 1920x1080
     
189. [ INFO ] Number of allocated frames: 3
     
190. [ INFO ] Resizable input with support of ROI crop and auto resize is disabled
     
191. 0.2FPS for (3 / 1) frames
     
192. Detection InferRequests usage: 0.0%
     
193. 
     
194. [ INFO ] Execution successful
     
195. 
     
196. 
     
197. ###################################################
     
198. 
     
199. Demo completed successfully.
     

复制代码

东方耀

注意：不要永久设置openvino的环境变量，会影响终端的其他conda环境的使用
比如：openvino的opencv就会影响 conda中的opencv-python

终端临时即可：source /home/jiang/intel/openvino/bin/setupvars.sh

东方耀

Operating Systems

    Ubuntu 18.04.x long-term support (LTS), 64-bit
    CentOS 7.4, 64-bit (for target only)
    Yocto Project v3.0, 64-bit (for target only and requires modifications)

东方耀

1. 运行例子：
   
2. 1、开新终端 source /home/jiang/intel/openvino/bin/setupvars.sh
   
3. 如果没有临时环境变量：
   
4. error while loading shared libraries: libinference_engine_transformations.so: cannot open shared object file
   
5. 2、去c++编译后的可执行文件所在目录：cd ~/inference_engine_samples_build/intel64/Release
   
6. 3、(CPU/GPU/MYRIAD)
   
7. ./classification_sample_async -i ~/intel/openvino/deployment_tools/demo/car.png -m ~/openvino_models/ir/public/squeezenet1.1/FP16/squeezenet1.1.xml -d CPU
   

复制代码

东方耀

1. Model Optimizer arguments:
   
2. Common parameters:
   
3.         - Path to the Input Model:         /home/jiang/py3_openvino_works/hyperlpr_py3_openvino/model_ori/mssd512_voc.caffemodel
   
4.         - Path for generated IR:         /home/jiang/py3_openvino_works/hyperlpr_py3_openvino/model_openvino/detect/FP32
   
5.         - IR output name:         mssd512_voc
   
6.         - Log level:         ERROR
   
7.         - Batch:         Not specified, inherited from the model
   
8.         - Input layers:         data
   
9.         - Output layers:         detection_out
   
10.         - Input shapes:         [1,3,512,512]
    
11.         - Mean values:         Not specified
    
12.         - Scale values:         Not specified
    
13.         - Scale factor:         Not specified
    
14.         - Precision of IR:         FP32
    
15.         - Enable fusing:         True
    
16.         - Enable grouped convolutions fusing:         True
    
17.         - Move mean values to preprocess section:         False
    
18.         - Reverse input channels:         False
    
19. Caffe specific parameters:
    
20.         - Path to Python Caffe* parser generated from caffe.proto:         /home/jiang/intel/openvino/deployment_tools/model_optimizer/mo/front/caffe/proto
    
21.         - Enable resnet optimization:         True
    
22.         - Path to the Input prototxt:         /home/jiang/py3_openvino_works/hyperlpr_py3_openvino/model_ori/mssd512_voc.prototxt
    
23.         - Path to CustomLayersMapping.xml:         Default
    
24.         - Path to a mean file:         Not specified
    
25.         - Offsets for a mean file:         Not specified
    
26. Model Optimizer version:        
    
27. [ WARNING ]  
    
28. Detected not satisfied dependencies:
    
29.         protobuf: installed: 3.0.0, required: >= 3.6.1
    
30. 
    
31. Please install required versions of components or use install_prerequisites script
    
32. /home/jiang/intel/openvino_2020.4.287/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites_caffe.sh
    
33. Note that install_prerequisites scripts may install additional components.
    
34. 
    
35. [ SUCCESS ] Generated IR version 10 model.
    
36. [ SUCCESS ] XML file: /home/jiang/py3_openvino_works/hyperlpr_py3_openvino/model_openvino/detect/FP32/mssd512_voc.xml
    
37. [ SUCCESS ] BIN file: /home/jiang/py3_openvino_works/hyperlpr_py3_openvino/model_openvino/detect/FP32/mssd512_voc.bin
    
38. [ SUCCESS ] Total execution time: 5.52 seconds.
    
39. [ SUCCESS ] Memory consumed: 83 MB.
    

复制代码

东方耀

推理引擎是一个支持C\C++和python的一套API接口，需要开发人员自己实现推理过程的开发，开发流程其实非常的简单，核心流程如下：

1、装载处理器的插件库
2、读取网络结构和权重
3、配置输入和输出参数
4、装载模型
5、创建推理请求
6、准备输入Data
7、推理
8、结果处理