... | ... | @@ -11,7 +11,7 @@ |
|
|
|
|
|
5.[Network training](manual-training)
|
|
|
- [Sessions](manual-training#sessions)
|
|
|
- [Input Manager ](manual-training#input-manager)
|
|
|
- [Input Manager](manual-training#input-manager)
|
|
|
- [Snapshots](manual-training#snapshots)
|
|
|
- [Console](manual-training#console)
|
|
|
- [Plotter](manual-training#plotter)
|
... | ... | @@ -33,32 +33,32 @@ Every session is represented by one entry in the list. Every item provides basic |
|
|
|
|
|
* **Remote Host** information: For remote sessions, the host name and connected port are displayed
|
|
|
* **Session ID**: A running ID of the session, this helps identifying which session a certain log-line in the console belongs to.
|
|
|
* **State Label**: A coloured marker indicating the session state. A Session can be in one of the follwoing states:
|
|
|
|
|
|
* **WAITING**: The session was just created and is ready to run
|
|
|
* **RUNNING**: The training process is running
|
|
|
* **PAUSED**: The session was paused and can be proceeded
|
|
|
* **FINISHED**: The training process was finished and the maximum iteration was reached
|
|
|
* **FAILED**: The session failed with some error, look in the error console for further information
|
|
|
* **INVALID**: Baristas internal checks found some faulty properties. More details are provided in an additional label instead of the Progress Bar. Even more Details are given when hovering over the latter label.
|
|
|
* **State Label**: A colored marker indicating the session state. A Session can be in one of the following states:
|
|
|
|
|
|
* **Ready**: The session was just created and is ready to run.
|
|
|
* **Running**: The training process is running.
|
|
|
* **Paused**: The session was paused and can be proceeded.
|
|
|
* **Finished**: The maximum iteration has been reached and the training process is thus finished.
|
|
|
* **Failed**: The session failed with some error. Look in the error console for further information.
|
|
|
* **Invalid**: Baristas internal checks found some faulty properties. More details are provided in an additional label instead of the Progress Bar. Even more Details are given when hovering over the latter label.
|
|
|
* **Not Connected**: A remote session lost its connection to a host. Make sure the remote machine is still running and its network connection is still active.
|
|
|
|
|
|
* **Snapshot Button**: When a session is running, the snapshot button can be used to create an unplanned (i.e. not defined in the solver properties) snapshot. Note: Your caffe version has to support the SIGHUP signal on Linux and Mac OS or SIGBREAK on Windows for this to work.
|
|
|
* **Delete Button**: This will delete a session and all its associated files and folders.
|
|
|
* **Context Menu**: More - less often used - functions like cloning and resetting sessions.
|
|
|
* **Context Menu**: More functionality like cloning and resetting sessions.
|
|
|
* **Play/Pause Button**: Start or Pause the training of a session.
|
|
|
* **Progress Bar**: Displays the iterations that have been trained so far and the maximum iterations as defined in the solver settings.
|
|
|
|
|
|
Remote sessions are not connected automatically, hence on loading a project, they have to be imported from the appropriate remote host, using the host manager.
|
|
|
|
|
|
Once training of a session has been started, the session can no longer be edited. This ensures that the training results are always in line with the displayed network and settings. If you want to change settings for a network for which training was already started, you can either create a new session which will have the same settings and alter them, or you can clone pre-learned weights from an existing session to a new session using the context menu in the old session. If you know, that you do not want to use the trained network state of one session, you could also select the reset option to throw away all training results and treat the session as new (Please note that you will loose all your training results for the selected session).
|
|
|
Once training of a session has been started, the session can no longer be edited. This ensures that the training results are always in line with the displayed network and settings. If you want to change settings for a network for which training was already started, you can either create a new session which will have the same settings and alter them, or you can clone pre-learned weights from an existing session to a new session using the context menu in the old session. If you know that you do not want to use the trained network state of one session, you could also select the reset option to throw away all training results and treat the session as new (Please note that you will lose all your training results for the selected session).
|
|
|
|
|
|
All files needed to train a network (except databases) and all files created during training are stored in the session folder. This is a sub-folder of the project directory or - for remote sessions - a subfolder of the session directory provided when starting the host. Hence, a session folder can be easily transfered to another machine where training could be resumed even if Barista is not installed.
|
|
|
All files needed to train a network (except databases) and all files created during training are stored in the session folder. This is a sub-folder of the project directory or - for remote sessions - a subfolder of the session directory provided when starting the host. Hence, a session folder can be easily transferred to another machine where training could be resumed even if Barista is not installed.
|
|
|
|
|
|
##Input Manager
|
|
|
## Input Manager
|
|
|
|
|
|
In order to save storage space, databases containing training and test data are not stored within a project or session folder. However, Barista offer an Input Manager that takes care of managing all your data.
|
|
|
The Input Manager can be accessed via manu bar: **Edit -> Input Manager** or pressing **Ctrl + I**.
|
|
|
In order to save storage space, databases containing training and test data are not stored within a project or session folder. However, Barista offers an Input Manager that takes care of managing all your data.
|
|
|
The Input Manager can be accessed via menu bar: **Edit -> Input Manager** or pressing **Ctrl + I**.
|
|
|
|
|
|
![InputManager](InputManager.png)
|
|
|
|
... | ... | @@ -74,12 +74,12 @@ For every database a unique ID is calculated by hashing its content. Hence loadi |
|
|
Every entry in the host manager has a number of manipulation options, from left to right the buttons are:
|
|
|
|
|
|
- Edit: Change the Name of the Database. This is only for user support, the database name has no influence on the training.
|
|
|
- Delete: Delete the Database entry. This does only delete the Reference in the Input Manager, not your actual data.
|
|
|
- Delete: Delete the Database entry. This only deletes the Reference in the Input Manager, but not the referenced files.
|
|
|
- Reload: Reload a database and its content. This can be used if a database became unavailable/dead because a host connection was lost or a network mount failed.
|
|
|
- Move Up/Down: Change the order of the items in the list.
|
|
|
- Assign to layer: Assign the data base as source to one of the available data layers in the network.
|
|
|
|
|
|
**HDF5** data base files can not directly be set as a source to a caffe HDF5 data layer. Barista supports creation and editing of the necessary txt files, that contain links to the actual HDF5 data files. Like for the databases themselfes, HDF5 files can be added to these files one-by-one, as a group or by recursively searching a directory.
|
|
|
**HDF5** data base files can not directly be set as a source to a caffe HDF5 data layer. Barista supports creation and editing of the necessary txt files, that contain links to the actual HDF5 data files. Like for the databases themselves, HDF5 files can be added to these files one-by-one, as a group or by recursively searching a directory. In each case, the relevant files are hashed again and the ID is updated.
|
|
|
|
|
|
## Snapshots
|
|
|
|
... | ... | @@ -91,23 +91,23 @@ It is possible to copy learned weights from one session to a new one to start tr |
|
|
To do so, click on the context-menu icon (three dots) for the session to be cloned in the session list and select **clone session**. You can then choose which snapshot you want to clone and a new session will be created in that state.
|
|
|
|
|
|
## Console
|
|
|
The console is the main output of barista. There are different callers, that register to the console and can write custom messages and errors. The messages should keep the user informed about the current state and actions. E.g. sessions write the caffe output here, the input manger would write errors if the selected data is faulty and other barista modules would also complain about errors.
|
|
|
The messages are not filtered at the beginning. They can be filtered by their type ("Text" or "Error") and caller. The filter is set by the two drop-down boxes.
|
|
|
If there is no filter set, every line has a prefix to identify the caller. it has the format: ``[HH:MM:ss, Caller] Sometext``
|
|
|
The console is the main output of barista. There are different callers, that register to the console and can write custom messages and errors. The messages should keep the user informed about the current state and actions. For example, sessions write the caffe output here, the input manger would write errors if the selected data is faulty, and other barista modules would also complain about errors.
|
|
|
The messages are not filtered by default. They can be filtered by their type ("Text" or "Error") and caller. The filter is set by the two drop-down boxes.
|
|
|
If there is no filter set, every line has a prefix to identify the caller. It has the format: ``[HH:MM:ss, Caller] Message``
|
|
|
|
|
|
## Plotter
|
|
|
The plotter is the main tool to visualize the current network performance during training. The caffe training log as seen in the console is parsed and the information are passed to the plotter. Every output (e.g. loss, accuracy, learning rate) can be selected to be plotted independently. Outputs named `loss` wil be added to the Plotter automatically, if a training session is started. As soon as a plotable key is written to the log of the caffe training process, it will appear in the settings panel of the plotter.
|
|
|
The plotter is the main tool to visualize the current network performance during training. The caffe training log as seen in the console is parsed and the information is passed to the plotter. Every output (e.g., loss, accuracy, learning rate) can be selected to be plotted independently. Outputs named `loss` will be added to the Plotter automatically, if a training session is started. As soon as a plottable key is written to the log of the caffe training process, it will appear in the settings panel of the plotter.
|
|
|
|
|
|
In the settings panel there is a list of all training sessions or loaded log files. Every session/log has a corresponding set of plotable keys represented as check boxes contained in the area on the right. Each key is associated to the train or test phase. If a check box is checked a plot line with the name 'logname.phase.key' is added to the plotter.
|
|
|
In the settings panel there is a list of all training sessions or loaded log files. Every session/log has a corresponding set of plottable keys represented as check boxes contained in the area on the right. Each key is associated to the train or test phase. If a key is enabled via the check box, a plot line with the name 'logname.phase.key' is added to the plotter.
|
|
|
|
|
|
The list of sessions and log files can be extended by starting a new session or opening an existing log file via the 'open file' button. Because the logs, phases and keys can be chosen separately the trainings can be easily compared and analysed.
|
|
|
The list of sessions and log files can be extended by starting a new session or opening an existing log file via the 'open file' button. Because the logs, phases and keys can be chosen separately the trainings can be easily compared and analyzed.
|
|
|
|
|
|
For instance one can compare the loss rate and the learning rate of a training in order to observe how the learning rate influences the loss rate. This can help to optimize the choice of the learning rate parameters.
|
|
|
For instance, one can compare the loss rate and the learning rate of a training in order to observe how the learning rate influences the training process and convergence. This can help to optimize the choice of the learning rate parameters.
|
|
|
|
|
|
Furthermore the user can choose how to plot the data (linear, logarithmic) and to plot it against the time or the number of iterations. Both options can be set in the settings panel.
|
|
|
Furthermore, the user can choose how to plot the data (linear, logarithmic) and to plot it against the time or the number of iterations. Both options can be set in the settings panel.
|
|
|
|
|
|
## CSV-export
|
|
|
It is possible to export the plotted data in the CSV-format. Only the selected data (which is actually plotted) will be written to a CSV-file. The file then contains one table for each session/log file and phase introduced by a comment for indetification. To export the plots just click on 'Export as CSV'. The exported file could look like this
|
|
|
It is possible to export the plotted data in the CSV-format. Only data which is currently selected for plotting will be written to a CSV-file. The file then contains one table for each session/log file and phase introduced by a comment for identification. To export the plots just click on 'Export as CSV'. The exported file could look similar to the following:
|
|
|
|
|
|
```
|
|
|
# Barista CSV-Export (timestamp)
|
... | ... | @@ -128,41 +128,38 @@ Iterations; Time; logname2.train.key1; logname2.train.key2; ... |
|
|
```
|
|
|
|
|
|
## Weight visualization
|
|
|
The weight plotter can visualize the filter trained in the neural network (currently limited to convolutional layer).
|
|
|
After splitting the convolutional layer into its filters, the matrices for every filter are rendered as a block of grayscale pixels.
|
|
|
The weight plotter can visualize the filters learned in the neural network (currently limited to convolutional layers).
|
|
|
After splitting a convolutional layer into its filters, the matrices for every filter are rendered as a block of grayscale pixels.
|
|
|
Very light pixels represent the weights with the higher parameter values, darker pixels the weights with the lower parameter values.
|
|
|
|
|
|
![image](/uploads/ea6a7099e4e1c3f2ee052233215fc810/image.png)
|
|
|
|
|
|
By use of the three comboboxes the desired session, snapshot and layer can be chosen respectively.
|
|
|
The desired session, snapshot and layer can be chosen via the three comboboxes.
|
|
|
By holding and dragging the left mouse button, the visualization can be panned in a desired direction.
|
|
|
Using the mouse wheel alters the scale of the picture.
|
|
|
A visualization can be saved by clicking the **save** button and entering a desired name and format-suffix.
|
|
|
|
|
|
A visualization can be saved to file by clicking the **save** button and entering a desired name and format-suffix.
|
|
|
|
|
|
## Deployment
|
|
|
Barista also supports deploying your network automatically. This is quite handy if you are not only interested in the training results themself, but want to use the final net e.g. inside of an external application, too. As the latter usually involves inference only, some parts of the created graph aren't needed anymore or need to be altered instead.
|
|
|
Barista also supports deploying your network automatically. This is quite handy if you are not only interested in the training results themselves, but also want to use the final net e.g. inside of an external application. As the latter usually involves inference only, some parts of the created graph aren't needed anymore or need to be altered instead.
|
|
|
|
|
|
To start the deployment process in Barista, use the according entry in the top menu bar.
|
|
|
|
|
|
Now you need to specify two things:
|
|
|
- a path pointing to an existing folder on your machine
|
|
|
- and a desired snapshot that has been created inside of your Barista project (or was imported by copying the file into one of the session folders)
|
|
|
- A path pointing to an existing folder on your machine
|
|
|
- The desired snapshot that has been created inside of your Barista project (or was imported by copying the file into one of the session folders)
|
|
|
|
|
|
The chosen snapshot determines which state (or weight set) of your network you want to deploy. The given path will be used as the destination to store all generated files. These files include a copy of the snapshot's caffemodel file, which consists of all trained weights, as well as a prototxt file containing a modified definition of your network's architecture.
|
|
|
The chosen snapshot determines which state (i.e., weight set) of your network you want to deploy. The given path will be used as the destination to store all generated files. These files include a copy of the snapshot's caffemodel file, which consists of all trained weights, as well as a prototxt file containing a modified definition of your network's architecture.
|
|
|
Before exporting those files, the following steps are performed automatically:
|
|
|
- All Data Layers are removed
|
|
|
- All layers requiring labeled data are removed, too (especially all Loss Layers as well as accuracy calculations etc.)
|
|
|
- Instead, new Input Layers with static input dimension will be added
|
|
|
- Input dimension will be determined automatically as well (as long as the former Data Layers used to have a valid data source)
|
|
|
- Blob connections are set automatically
|
|
|
- Usually, there will be only one Input Layer. However, note that, the number of newly-added input layers might be lower than the number of previously-removed data layers. On the one hand, a new input layer will be added only once for every data blob name, while multiple data layers might have used the same data blob name (in Training and Test phase). On the other hand, input layers will only be added, if at least one other layer is using the provided data.
|
|
|
- All Data Layers are removed.
|
|
|
- All layers requiring labeled data are removed (including all Loss Layers, accuracy calculations, etc.).
|
|
|
- Instead, new Input Layers with static input dimension will be added.
|
|
|
- Input dimension will be determined automatically (as long as the former Data Layers used to have a valid data source)
|
|
|
- Blob connections are set automatically.
|
|
|
- Usually, there will be only one Input Layer. However, note that the number of newly-added input layers might be lower than the number of previously-removed data layers. On the one hand, a new input layer will be added only once for every data blob name, while multiple data layers might have used the same data blob name (in Training and Test phase). On the other hand, input layers will only be added if at least one other layer is using the provided data.
|
|
|
- Append a new Softmax layer (only if none does exist yet and a SoftmaxWithLoss layer was used before).
|
|
|
|
|
|
The above described rules are based on the information provided in the following caffe wiki page: https://github.com/BVLC/caffe/wiki/Using-a-Trained-Network:-Deploy
|
|
|
The above described rules are based on the information provided in the following caffe wiki page: https://github.com/BVLC/caffe/wiki/Using-a-Trained-Network:-Deploy
|
|
|
|
|
|
Finally, some restrictions apply to the deployment process:
|
|
|
- At least the label blob name must follow the naming conventions (so it's always called "label"). Raises a warning.
|
|
|
- At least the label blob name must follow the naming conventions and thus has to be called "label".
|
|
|
- The shape of a Data layer can only be determined automatically, if the layer type is either "Data" (LMDB or LEVELDB) or "HDF5Data". Otherwise, a warning will inform the user about necessary manual changes.
|
|
|
|
|
|
| [previous page](manual-network) | [start page](manual) | [next page](manual-miscellaneous) | |
|
|
\ No newline at end of file |