Training of a random model.

Post a reply


This question is a means of preventing automated form submissions by spambots.
Smilies
:D :) ;) :( :o :shock: :? 8-) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :!: :?: :idea: :arrow: :| :mrgreen: :geek: :ugeek:

BBCode is ON
[img] is ON
[flash] is OFF
[url] is ON
Smilies are ON

Topic review
   

Expand view Topic review: Training of a random model.

OpenCV Artificial Neural Network Model

Post by Creator » Sun Sep 24, 2017, 17:33

Artificial Neural Network (ANN) classifier is inspired by the biological neural networks that constitute mammal brains. The original goal of such approaches was to solve problems in the same way that a human brain would. Hence, ANN classifiers learn (or progressively improve performance) by considering training samples, generally without class-specific features.

An ANN is based on a collection of connected units (neurons). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving neuron can process the signals and then signal downstream neurons connected to it. Neurons and synapses have weights that vary as learning proceeds and control the strength of the signal that is sent downstream. Typically, neurons are organized in layers, where different layers may perform different kinds of transformations on their inputs. Signals travel from the first (input), to the last (output) layer, possibly after traversing the layers multiple times.

We initialize the number of neurons in the input layer to be equal to the number of features m (m = 2 for this example) and the number of neurons in the output layer to be equal to the number of labels k (k = 3 for this example). In order to represent the ANN decision on the correct label into the form of the posterior probability, we take the k scaled signals leaving the neurons from the output layer and use these signals’ strength directly for filling the needed potentials (see Fig. 11).

PDF[featureVector][state] = CvANN(featureVector)

hist2d_CvANN.jpg
Fig. 11 (256 x 256 image, represents 3 restored PDF via OpenCV Artificial Neural Network model for 3 classes on 2D feature space)
hist2d_CvANN.jpg (11.32 KiB) Viewed 9114 times

OpenCV k-Nearest Neighbors Model

Post by Creator » Wed Aug 23, 2017, 01:31

The same as above, it is also possible to make use of the OpenCV implementation of the GMM. It is based on the Expectation Maximization (EM) method and produces the results, depicted at Fig. 10 (Each class is approximated with 16 Gaussians, default parameters).

PDF[featureVector][state] = CvKNN(featureVector).

hist2d_CvKNN.jpg
Fig. 10 (256 x 256 image, represents 3 restored PDF via OpenCV k-Nearest Neighbors model for 3 classes on 2D feature space)
hist2d_CvKNN.jpg (14.79 KiB) Viewed 31524 times

K-Nearest Neighbors Model

Post by Creator » Wed May 10, 2017, 08:55

A new discriminative classifier is based on k-nearest neighbors algorithm (KNN), where the input consists of the k closest training samples in the feature space and the output depends on k-Nearest Neighbors. Its results for our test setup are depicted at Fig. 9. As the another discriniative methods (Random Forests, etc.) it provides high potentials for almost all of the samples, including those, which are very distant from the training samples. In order to organize the training samples in the k-D tree data structure, the algorithm also needs keeping all them in memory. However it has incredibly good performance on low-dimentional feature spaces.

PDF[featureVector][state] = KNN(featureVector).

hist2d_KNN.jpg
Fig. 9 (256 x 256 image, represents 3 restored PDF via k-Nearest Neighbors model for 3 classes on 2D feature space)
hist2d_KNN.jpg (27.11 KiB) Viewed 31714 times

Microsoft Sherwood Random Forest Model

Post by Creator » Fri Jun 07, 2013, 22:04

Another Random Forest implementation is taken from the Microsoft Sherwood library. Its results for our test setup are depicted at Fig. 8. These results are not much differ from the results, depicted in the Fig. 7, nevertheless, the classification accuracy may be different from dataset to dataset.

PDF[featureVector][state] = Forest(featureVector).

hist2d_MsRF.jpg
Fig. 8 (256 x 256 image, represents 3 restored PDF via Microsoft random forest model for 3 classes on 2D feature space)
hist2d_MsRF.jpg (31.68 KiB) Viewed 40201 times

OpenCV Random Forest Model

Post by Creator » Fri Jan 25, 2013, 15:50

One more example of OpenCV training approach - Random Forest (RF). Its results for our test setup are depicted at Fig. 7. It has the same drawback as OpenCV GMM approach - all the training samples must be kept in memory for training. It is also very slow, but it is shown to produce good classification results in spite of the Fig. 7 differs from the Fig. 1 wery much (because of the discriminative nature of random forest approach).

PDF[featureVector][state] = CvRTrees_predictor_state(featureVector) .

hist2d_RF.jpg
Fig. 7 (256 x 256 image, represents 3 restored PDF via OpenCV random forest model for 3 classes on 2D feature space)
hist2d_RF.jpg (43.7 KiB) Viewed 40760 times

OpenCV Gaussian Mixture Model

Post by Creator » Tue Dec 25, 2012, 19:08

The same as above, it is also possible to make use of the OpenCV implementation of the GMM. It is based on the Expectation Maximization (EM) method and produces the results, depicted at Fig. 6 (Each class is approximated with 16 Gaussians, default parameters).
In comparison to our sequential GMM approach, the OpenCV implementation has the following drawbacks:
  • OpenCV GMM is about 10 times slower
  • All the training samples must be kept in memory for training => this leads to impossibility of training the model on large training dataset, when the PC RAM resource is bounded.
  • Relatively poor accuracy
PDF[featureVector][state] = CvEMpredictor_state(featureVector) .

hist2d_CvGMM.jpg
Fig. 6 (256 x 256 image, represents 3 restored PDF via OpenCV gaussian mixture model for 3 classes on 2D feature space)
hist2d_CvGMM.jpg (19.96 KiB) Viewed 40865 times

Gaussian Mixture Model

Post by Creator » Tue Oct 23, 2012, 17:06

In spite of the Gaussian Model can encode the inter-feature dependences, it may produce even worse results as Bayes model. Approximating a complex form of real distributions is sometimes almost impossible with a single Gaussian function. In that reason we can extend this model by substituting a single Gaussian with an additive superposition of several Gaussians functions:

PDF[featureVector][state] = SUM_{g \in nGaussians[state]} (k[g] * G[state][g](featureVector)), where nGaussians[state] - nuber of Gaussian functions for approximation of the PDF of the state state, and k[g] - is a weight koefficient, whith SUM_{g \in nGaussians[state]} (k[g]) = 1.

hist2d_GMM.jpg
Fig. 5 (256 x 256 image, represents 3 restored PDF via gaussian mixture model for 3 classes on 2D feature space)
hist2d_GMM.jpg (19.69 KiB) Viewed 41058 times

This approach allows us to shrink nStates * 256^nFeatures data values into nStates * nGaussians * (nFeatures^2 + nFeatures) data cells. It is also quadratic under the nFeatures.

Gaussian Model

Post by Creator » Thu Jun 21, 2012, 01:13

Using Bayes Model in training we gain high performance. Nevertheless we lose all the inter-feature dependencies, i.e. each feature influences the resulting potential independently from all other features. It is also possible to use approximation, which is free from that drawback, e.g. approximation of the original PDFs with Gaussian functions. In this case, the inter-features dependencies are coded in covariance matrix – one of two parameters of a multi-dimensional Gaussian kernel:

PDF[featureVector][state] = G[state](featureVector), where G[](x) is a nFeatures-dimensional Gaussian function.

The restored normalized PDFs are depicted at Fig. 4 for Gaussian model.

hist2d_GM.jpg
Fig. 4 (256 x 256 image, represents 3 restored PDF via gaussian model for 3 classes on 2D feature space)
hist2d_GM.jpg (23.64 KiB) Viewed 41213 times


This approach allows us to shrink nStates * 256^nFeatures data values into nStates * (nFeatures^2 + nFeatures) data cells. As we can see it is quadratic under the nFeatures.

Bayes Model

Post by Creator » Wed Jun 20, 2012, 23:28

In order to reconstruct the ortogonal n-dimensional PDF function from n one-dimensional PDFs, we make use of the superposition of them:

PDF[featureVector][state] = MUL_{feature \in featureVector} (H[feature][state]);

Code: Select all

for (int state = 0; state < nStates; state++) {
   PDF[state] = 1;
   for (int feature = 0; feature < nFeatures; feature++) {   
      byte featureValue = featureVector[feature];
      if (H[feature][state].n != 0)
         PDF[state] *= H[feature][state].data[featureValue] / H[feature][state].n;
      else
         PDF[state] = 0;
   }
}


The restored normalized PDFs are depicted at Fig. 3.

hist2d_MUL.jpg
Fig. 3 (256 x 256 image, represents 3 restored PDF via multiplicative superpositoon for 3 classes on 2D feature space
hist2d_MUL.jpg (26.09 KiB) Viewed 41214 times

Bayes Model

Post by Creator » Tue Jun 19, 2012, 02:45

The Bayes model approximates the PDFs via decomposing n-Dimensional space into n one-Dimensional signals. For this purpose we can build the 1-Dimensional PDFs for each feature and for each state (class), neglecting all the dependensies between features. These 1-Dimensional PDFs are histohrams H[feature][state] of feature feature occurances for the given classes state. These normalized histogramms of length 256 are presented on Fig. 2

hist.jpg
Fig. 2 (6 histograms of length 256, represent 1D PFDs for 3 classes on 2D feature space)

This approach allows us to shrink nStates * 256^nFeatures data values into nStates * 256 * nFeatures data cells. As expected, all the features correlation information will be lost.

Top