Gitiles
Code Review Sign In
gerrit-public.fairphone.software / platform / external / ComputeLibrary / 67c8c91522e5be8156b77f57e63c0253535c902a / . / documentation / data_import.xhtml
blob: fcf382c08007a7099f6834edd9c33390b9574ed6 [file] [log] [blame]
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]1<!-- HTML header for doxygen 1.8.9.1-->
2<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
3<html xmlns="http://www.w3.org/1999/xhtml">
4<head>
5<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
6<meta http-equiv="X-UA-Compatible" content="IE=9"/>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]7<meta name="generator" content="Doxygen 1.8.11"/>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]8<meta name="robots" content="NOINDEX, NOFOLLOW" /> <!-- Prevent indexing by search engines -->
9<title>Compute Library: Importing data from existing models</title>
10<link href="tabs.css" rel="stylesheet" type="text/css"/>
11<script type="text/javascript" src="jquery.js"></script>
12<script type="text/javascript" src="dynsections.js"></script>
13<link href="navtree.css" rel="stylesheet" type="text/css"/>
14<script type="text/javascript" src="resize.js"></script>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]15<script type="text/javascript" src="navtreedata.js"></script>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]16<script type="text/javascript" src="navtree.js"></script>
17<script type="text/javascript">
18 $(document).ready(initResizable);
19 $(window).load(resizeHeight);
20</script>
21<link href="search/search.css" rel="stylesheet" type="text/css"/>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]22<script type="text/javascript" src="search/searchdata.js"></script>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]23<script type="text/javascript" src="search/search.js"></script>
24<script type="text/javascript">
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]25 $(document).ready(function() { init_search(); });
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]26</script>
27<script type="text/x-mathjax-config">
28 MathJax.Hub.Config({
29 extensions: ["tex2jax.js"],
30 jax: ["input/TeX","output/HTML-CSS"],
31});
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]32</script><script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js"></script>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]33<link href="doxygen.css" rel="stylesheet" type="text/css" />
34</head>
35<body>
36<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
37<div id="titlearea">
38<table cellspacing="0" cellpadding="0">
39 <tbody>
40 <tr style="height: 56px;">
41 <td style="padding-left: 0.5em;">
42 <div id="projectname">Compute Library
Jenkinsc3f34a42018-03-02 12:38:09 +0000[diff] [blame]43 &#160;<span id="projectnumber">18.03</span>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]44 </div>
45 </td>
46 </tr>
47 </tbody>
48</table>
49</div>
50<!-- end header part -->
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]51<!-- Generated by Doxygen 1.8.11 -->
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]52<script type="text/javascript">
53var searchBox = new SearchBox("searchBox", "search",false,'Search');
54</script>
55 <div id="navrow1" class="tabs">
56 <ul class="tablist">
57 <li><a href="index.xhtml"><span>Main&#160;Page</span></a></li>
58 <li class="current"><a href="pages.xhtml"><span>Related&#160;Pages</span></a></li>
59 <li><a href="namespaces.xhtml"><span>Namespaces</span></a></li>
60 <li><a href="annotated.xhtml"><span>Data&#160;Structures</span></a></li>
61 <li><a href="files.xhtml"><span>Files</span></a></li>
62 <li>
63 <div id="MSearchBox" class="MSearchBoxInactive">
64 <span class="left">
65 <img id="MSearchSelect" src="search/mag_sel.png"
66 onmouseover="return searchBox.OnSearchSelectShow()"
67 onmouseout="return searchBox.OnSearchSelectHide()"
68 alt=""/>
69 <input type="text" id="MSearchField" value="Search" accesskey="S"
70 onfocus="searchBox.OnSearchFieldFocus(true)"
71 onblur="searchBox.OnSearchFieldFocus(false)"
72 onkeyup="searchBox.OnSearchFieldChange(event)"/>
73 </span><span class="right">
74 <a id="MSearchClose" href="javascript:searchBox.CloseResultsWindow()"><img id="MSearchCloseImg" border="0" src="search/close.png" alt=""/></a>
75 </span>
76 </div>
77 </li>
78 </ul>
79 </div>
80</div><!-- top -->
81<div id="side-nav" class="ui-resizable side-nav-resizable">
82 <div id="nav-tree">
83 <div id="nav-tree-contents">
84 <div id="nav-sync" class="sync"></div>
85 </div>
86 </div>
87 <div id="splitbar" style="-moz-user-select:none;"
88 class="ui-resizable-handle">
89 </div>
90</div>
91<script type="text/javascript">
92$(document).ready(function(){initNavTree('data_import.xhtml','');});
93</script>
94<div id="doc-content">
95<!-- window showing the filter options -->
96<div id="MSearchSelectWindow"
97 onmouseover="return searchBox.OnSearchSelectShow()"
98 onmouseout="return searchBox.OnSearchSelectHide()"
99 onkeydown="return searchBox.OnSearchSelectKey(event)">
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]100</div>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]101
102<!-- iframe showing the search results (closed by default) -->
103<div id="MSearchResultsWindow">
104<iframe src="javascript:void(0)" frameborder="0"
105 name="MSearchResults" id="MSearchResults">
106</iframe>
107</div>
108
109<div class="header">
110 <div class="headertitle">
111<div class="title">Importing data from existing models </div> </div>
112</div><!--header-->
113<div class="contents">
114<div class="toc"><h3>Table of Contents</h3>
115<ul><li class="level1"><a href="#caffe_data_extractor">Extract data from pre-trained caffe model</a><ul><li class="level2"><a href="#caffe_how_to">How to use the script</a></li>
116<li class="level2"><a href="#caffe_result">What is the expected output from the script</a></li>
117</ul>
118</li>
119<li class="level1"><a href="#tensorflow_data_extractor">Extract data from pre-trained tensorflow model</a><ul><li class="level2"><a href="#tensorflow_how_to">How to use the script</a></li>
120<li class="level2"><a href="#tensorflow_result">What is the expected output from the script</a></li>
121</ul>
122</li>
123</ul>
124</div>
125<div class="textblock"><h1><a class="anchor" id="caffe_data_extractor"></a>
126Extract data from pre-trained caffe model</h1>
127<p>One can find caffe <a href="https://github.com/BVLC/caffe/wiki/Model-Zoo">pre-trained models</a> on caffe's official github repository.</p>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]128<p>The <a class="el" href="caffe__data__extractor_8py.xhtml">caffe_data_extractor.py</a> provided in the <a class="el" href="dir_53e6fa9553ac22a5646d2a2b2d7b97a1.xhtml">scripts</a> folder is an example script that shows how to extract the parameter values from a trained model.</p>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]129<dl class="section note"><dt>Note</dt><dd>complex networks might require altering the script to properly work.</dd></dl>
130<h2><a class="anchor" id="caffe_how_to"></a>
131How to use the script</h2>
132<p>Install caffe following <a href="http://caffe.berkeleyvision.org/installation.html">caffe's document</a>. Make sure the pycaffe has been added into the PYTHONPATH.</p>
133<p>Download the pre-trained caffe model.</p>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]134<p>Run the <a class="el" href="caffe__data__extractor_8py.xhtml">caffe_data_extractor.py</a> script by </p><pre class="fragment"> python caffe_data_extractor.py -m &lt;caffe model&gt; -n &lt;caffe netlist&gt;
135</pre><p>For example, to extract the data from pre-trained caffe Alex model to binary file: </p><pre class="fragment"> python caffe_data_extractor.py -m /path/to/bvlc_alexnet.caffemodel -n /path/to/caffe/models/bvlc_alexnet/deploy.prototxt
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]136</pre><p>The script has been tested under Python2.7.</p>
137<h2><a class="anchor" id="caffe_result"></a>
138What is the expected output from the script</h2>
139<p>If the script runs successfully, it prints the names and shapes of each layer onto the standard output and generates *.npy files containing the weights and biases of each layer.</p>
140<p>The <a class="el" href="namespacearm__compute_1_1utils.xhtml#af214346f90d640ac468dd90fa2a275cc">arm_compute::utils::load_trained_data</a> shows how one could load the weights and biases into tensor from the .npy file by the help of Accessor.</p>
141<h1><a class="anchor" id="tensorflow_data_extractor"></a>
142Extract data from pre-trained tensorflow model</h1>
143<p>The script <a class="el" href="tensorflow__data__extractor_8py.xhtml">tensorflow_data_extractor.py</a> extracts trainable parameters (e.g. values of weights and biases) from a trained tensorflow model. A tensorflow model consists of the following two files:</p>
144<p>{model_name}.data-{step}-{global_step}: A binary file containing values of each variable.</p>
145<p>{model_name}.meta: A binary file containing a MetaGraph struct which defines the graph structure of the neural network.</p>
146<dl class="section note"><dt>Note</dt><dd>Since Tensorflow version 0.11 the binary checkpoint file which contains the values for each parameter has the format of: {model_name}.data-{step}-of-{max_step} instead of: {model_name}.ckpt When dealing with binary files with version &gt;= 0.11, only pass {model_name} to -m option; when dealing with binary files with version &lt; 0.11, pass the whole file name {model_name}.ckpt to -m option.</dd>
147<dd>
148This script relies on the parameters to be extracted being in the 'trainable_variables' tensor collection. By default all variables are automatically added to this collection unless specified otherwise by the user. Thus should a user alter this default behavior and/or want to extract parameters from other collections, tf.GraphKeys.TRAINABLE_VARIABLES should be replaced accordingly.</dd></dl>
149<h2><a class="anchor" id="tensorflow_how_to"></a>
150How to use the script</h2>
151<p>Install tensorflow and numpy.</p>
152<p>Download the pre-trained tensorflow model.</p>
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]153<p>Run <a class="el" href="tensorflow__data__extractor_8py.xhtml">tensorflow_data_extractor.py</a> with </p><pre class="fragment"> python tensorflow_data_extractor -m &lt;path_to_binary_checkpoint_file&gt; -n &lt;path_to_metagraph_file&gt;
154</pre><p>For example, to extract the data from pre-trained tensorflow Alex model to binary files: </p><pre class="fragment"> python tensorflow_data_extractor -m /path/to/bvlc_alexnet -n /path/to/bvlc_alexnet.meta
155</pre><p>Or for binary checkpoint files before Tensorflow 0.11: </p><pre class="fragment"> python tensorflow_data_extractor -m /path/to/bvlc_alexnet.ckpt -n /path/to/bvlc_alexnet.meta
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]156</pre><dl class="section note"><dt>Note</dt><dd>with versions &gt;= Tensorflow 0.11 only model name is passed to the -m option</dd></dl>
157<p>The script has been tested with Tensorflow 1.2, 1.3 on Python 2.7.6 and Python 3.4.3.</p>
158<h2><a class="anchor" id="tensorflow_result"></a>
159What is the expected output from the script</h2>
160<p>If the script runs successfully, it prints the names and shapes of each parameter onto the standard output and generates .npy files containing the weights and biases of each layer.</p>
161<p>The <a class="el" href="namespacearm__compute_1_1utils.xhtml#af214346f90d640ac468dd90fa2a275cc">arm_compute::utils::load_trained_data</a> shows how one could load the weights and biases into tensor from the .npy file by the help of Accessor. </p>
162</div></div><!-- contents -->
163</div><!-- doc-content -->
164<!-- start footer part -->
165<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
166 <ul>
Jenkinsc3f34a42018-03-02 12:38:09 +0000[diff] [blame]167 <li class="footer">Generated on Fri Mar 2 2018 12:37:56 for Compute Library by
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]168 <a href="http://www.doxygen.org/index.html">
Anthony Barbier8140e1e2017-12-14 23:48:46 +0000[diff] [blame]169 <img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.11 </li>
Kaizen8938bd32017-09-28 14:38:23 +0100[diff] [blame]170 </ul>
171</div>
172</body>
173</html>