Download soft/USE it/receive help and feedback
Types Of Application Software
With Examples
Algorithms developed and used by our team in the research
Read and ask questions.
Regression models estimate the mathematical relationship between a set of independent
variables, arranged in a data matrix X, and a quantitative independent variable, which
is collected in the response column vector y. Linear regression methods estimate this
relationship with a linear combination of the independent variables
variables, arranged in a data matrix X, and a quantitative independent variable, which
is collected in the response column vector y. Linear regression methods estimate this
relationship with a linear combination of the independent variables
Ordinary least squares method provides the regression coecients that minimize the
Residual Sum of Squares (RSS). OLS is denitely one of the most common and popular regression approaches. However, there are some limitations related to the constraints of the inversion of the information matrix
Residual Sum of Squares (RSS). OLS is denitely one of the most common and popular regression approaches. However, there are some limitations related to the constraints of the inversion of the information matrix
Unlike OLS, ridge regression (RR) can handle the presence of highly correlated variables.
The main idea of RR is to add a k parameter to the diagonal of the matrix (XTX) to
mitigate the inversion problem due to illconditioned data matrices. The RR regression
coefficients are calculated by the following equation:
The main idea of RR is to add a k parameter to the diagonal of the matrix (XTX) to
mitigate the inversion problem due to illconditioned data matrices. The RR regression
coefficients are calculated by the following equation:
Principal Component Regression integrates Principal Component Analysis (PCA) modelling to OLS regression. PCR coecients are estimated in the following way. The optimal number M of principal components to be used in the regression model can be selected by a cross-validation protocol.
Similarly to PCR, PLS regression method identies a new set of features, known as
Latent Variables(LVs), which are used to estimate the regression coecients. PLS nds LV directions that describe the largest amount of data variance in X and are most correlated to the response. There are several dierent algorithms to estimate the PLS coefficients the regression toolbox implements the NIPALS algorithm
Latent Variables(LVs), which are used to estimate the regression coecients. PLS nds LV directions that describe the largest amount of data variance in X and are most correlated to the response. There are several dierent algorithms to estimate the PLS coefficients the regression toolbox implements the NIPALS algorithm
The k-Nearest Neighbours (kNN) approach was originally proposed for supervised classiffication but later adapted to multivariate regression. The response of a target is estimated as the average of the responses of the k nearest neighbours, that is, the k most similar training samples, which are identied by a selected metric. The Binned Nearest Neighbours (BNN) approach is quite similar to kNN, with the dierence that the prediction is based on a flexible number of neighbours. Similarity intervals (i.e., bins) are dened on the basis of a tuning parameter alpha and the training samples are then distributed into these intervals according to their similarity to the target. All the neighbours falling into the bin with the largest similarity are then considered for the prediction. The optimal number of k neighbours or the optimal value of the alpha tuning parameters are selected by crossvalidation protocols.
Variable selection approaches aim to find a subset of variables that can improve the
prediction performance and/or simplify the model. This toolbox includes four selection
strategies (All Subset models, Forward Selection, Genetic Algorithms and Reshaped Sequential Replacement), which can be coupled with any of the regression methods previously described. For all selection methods, the RMSECV and the determination coefficient R2 in cross-validation are used as the fitness functions.
prediction performance and/or simplify the model. This toolbox includes four selection
strategies (All Subset models, Forward Selection, Genetic Algorithms and Reshaped Sequential Replacement), which can be coupled with any of the regression methods previously described. For all selection methods, the RMSECV and the determination coefficient R2 in cross-validation are used as the fitness functions.
This approach evaluates all the possible combinations of the J variables and guarantees
that the best subset of variables is found by minimising the regression error in crossvalidation (RMSECV); however, it is highly computationally consuming and thus it
becomes unsuitable for large numbers of variables.
that the best subset of variables is found by minimising the regression error in crossvalidation (RMSECV); however, it is highly computationally consuming and thus it
becomes unsuitable for large numbers of variables.
This is a basic and simple approach: starting from an empty set, variables are sequentially
added to this set, one at a time; in each iteration, the criterion to select which variable to
include is based on the minimisation of RMSECV. In this approach, results can be biased
by the rst variables included in the selected set and, when a high number of variables is
present, the application of this method is not suggested.
added to this set, one at a time; in each iteration, the criterion to select which variable to
include is based on the minimisation of RMSECV. In this approach, results can be biased
by the rst variables included in the selected set and, when a high number of variables is
present, the application of this method is not suggested.
Genetic Algorithms (GAs) are a popular variable selection approach that is inspired by the natural selection of the individuals of a population during its evolution. In the GA framework, each gene represents a variable and each chromosome (sequence of genes/variables) represents a model. The evolution of the population (the default population consists of 30 chromosomes, that is, individuals) is determined by two processes: in the crossover step, pairs of chromosomes generate new individuals according to a crossover probability (default 50%), while in the mutation step genes of each chromosome can change according to a mutation probability (default 1%). When a new chromosome has better performance than that of the already existing ones, it enters the population and the worst model is discarded.
After evolution, difierent strategies can be used to select the fiinal subset of variables. In this toolbox, the GA evolution is independently repeated for a fixed number of times (called runs) and the relative occurrence frequency of each variable in the best models is calculated. Then, the user can set a frequency threshold and only the variables with
occurrence frequencies higher than the threshold will be further processed by a forward selection or All Subset models procedure. Finally, a list of models (including the best subsets of selected variables) is proposed, from which the user can choose the preferred model.
After evolution, difierent strategies can be used to select the fiinal subset of variables. In this toolbox, the GA evolution is independently repeated for a fixed number of times (called runs) and the relative occurrence frequency of each variable in the best models is calculated. Then, the user can set a frequency threshold and only the variables with
occurrence frequencies higher than the threshold will be further processed by a forward selection or All Subset models procedure. Finally, a list of models (including the best subsets of selected variables) is proposed, from which the user can choose the preferred model.
M5 model tree is a decision tree learner for regression task, which is a binary decision
tree having linear regression functions at the terminal (leaf) nodes, which can predict
continuous numerical attributes. M5 model tree can simulate the phenomena with very
high dimensionality up to hundreds of attributes. This ability sets M5 apart from
regression tree learners at the time (like MARS), whose computational cost grows very
quickly when the number of features increases A model tree generation includes two
different steps.
The first step involves using a splitting criterion to make a decision tree. The splitting
criterion of M5 model tree algorithm is based on treating the standard deviation of the
class values which reach a node as an error measure at that node, and calculating the
expected reduction in this error as a result of testing each attribute at that node.
By splitting process, the data in child nodes get lower standard deviation values compared to
the parent node. After evaluating all possible splits, M5 selects the split which maximizes
the expected error reduction. This division often produces a large tree-like structure that
may cause over tting. Consequently, the tree must be pruned back.
The second stage would involve pruning the overgrown tree and replacing the subtrees
with linear regression functions. This technique of generating the model tree splits the
parameter space into subspaces and builds a linear regression model in each of them.
With method M5 can build trees and decision rules either individually or in
ensembles and test them on separate test sets or using Cross-Validation, use them for
prediction, assess variable importance, decompose their predictions into input variable
contributions, as well as print and plot the structure.
tree having linear regression functions at the terminal (leaf) nodes, which can predict
continuous numerical attributes. M5 model tree can simulate the phenomena with very
high dimensionality up to hundreds of attributes. This ability sets M5 apart from
regression tree learners at the time (like MARS), whose computational cost grows very
quickly when the number of features increases A model tree generation includes two
different steps.
The first step involves using a splitting criterion to make a decision tree. The splitting
criterion of M5 model tree algorithm is based on treating the standard deviation of the
class values which reach a node as an error measure at that node, and calculating the
expected reduction in this error as a result of testing each attribute at that node.
By splitting process, the data in child nodes get lower standard deviation values compared to
the parent node. After evaluating all possible splits, M5 selects the split which maximizes
the expected error reduction. This division often produces a large tree-like structure that
may cause over tting. Consequently, the tree must be pruned back.
The second stage would involve pruning the overgrown tree and replacing the subtrees
with linear regression functions. This technique of generating the model tree splits the
parameter space into subspaces and builds a linear regression model in each of them.
With method M5 can build trees and decision rules either individually or in
ensembles and test them on separate test sets or using Cross-Validation, use them for
prediction, assess variable importance, decompose their predictions into input variable
contributions, as well as print and plot the structure.
Model diagnostics
Depending on the regression method, several diagnostic tools are provided to assess the
model quality, evaluate the model assumptions, investigate whether or not there are
observations with a large inuence on the model and identify critical points, such as
the outliers. The most common dispersion and correlation-based measures to evaluate the
overall model quality, that is, the root mean squared error (RMSE) and the coecient of
determination R2, are calculated as:
model quality, evaluate the model assumptions, investigate whether or not there are
observations with a large inuence on the model and identify critical points, such as
the outliers. The most common dispersion and correlation-based measures to evaluate the
overall model quality, that is, the root mean squared error (RMSE) and the coecient of
determination R2, are calculated as:
Module description that allows the application of several methods
for variable reduction based on correlation analysis
for variable reduction based on correlation analysis
This method is based on a simple algorithm that
searches for couples of variables with a pairwise correlation higher than a given threshold
(set by default to 0.95); when a couple is selected, the variable with the highest average
value of absolute correlation (with all the remaining variables) is chosen and put in the
ranking.
searches for couples of variables with a pairwise correlation higher than a given threshold
(set by default to 0.95); when a couple is selected, the variable with the highest average
value of absolute correlation (with all the remaining variables) is chosen and put in the
ranking.
The K Inflation factor (KIF) is based on the K multivariate correlation index.
In this way, the variable with the highest multivariate correlation with those remaining
is eliminated: in fact, this means that when the variable q is excluded from the data, the
remaining multivariate correlation derived from the remaining p−1 variables is maximally
decreased.
In this way, the variable with the highest multivariate correlation with those remaining
is eliminated: in fact, this means that when the variable q is excluded from the data, the
remaining multivariate correlation derived from the remaining p−1 variables is maximally
decreased.
When using PCA, the loading coecients of the first principal component can represent
a measure of the capability of each variable to contribute to the whole correlation.
Therefore, the loadings of the first principal component can be used for ranking the
variables, from the most correlated (high loadings) to the less correlated (low loadings,
i.e. loading values close to zero). Then, a correlation ranking of all the p descriptors is
obtained, where the highest loadings correspond to the descriptors mostly correlated to the remaining ones.
Once the correlation matrix has been calculated, this approach requires only one eigenvalue/eigenvector estimate, since it adopts a non-iterative procedure. A further development of this approach consists in performing the PCA calculation iteratively, excluding in each step the variable with the highest loading on the first principal component.
a measure of the capability of each variable to contribute to the whole correlation.
Therefore, the loadings of the first principal component can be used for ranking the
variables, from the most correlated (high loadings) to the less correlated (low loadings,
i.e. loading values close to zero). Then, a correlation ranking of all the p descriptors is
obtained, where the highest loadings correspond to the descriptors mostly correlated to the remaining ones.
Once the correlation matrix has been calculated, this approach requires only one eigenvalue/eigenvector estimate, since it adopts a non-iterative procedure. A further development of this approach consists in performing the PCA calculation iteratively, excluding in each step the variable with the highest loading on the first principal component.
After the cycle is ended, a VIF value will be calculated for each of the p variables. As
the index represent the explained variance of the model, it also points out how much y is
correlated with the other variables in X. For this reason, the variable with the highest
VIF value is the most correlated to the others, consequently it is put in the nal ranking.
Once the selected variable is removed from the dataset, the algorithm proceed in the same way for the remaining ones; it can be performed calculating again the VIF values at each step (iterative version) or it can proceed using the values calculated the first time.
the index represent the explained variance of the model, it also points out how much y is
correlated with the other variables in X. For this reason, the variable with the highest
VIF value is the most correlated to the others, consequently it is put in the nal ranking.
Once the selected variable is removed from the dataset, the algorithm proceed in the same way for the remaining ones; it can be performed calculating again the VIF values at each step (iterative version) or it can proceed using the values calculated the first time.
Those two methods, originally proposed by Jolie, are based on the Principal Component
Analysis of the dataset. The B2 method consists in a sequential analysis of all the Principal Components (PC), starting from the last one (the less significant). For each PC, the first not already chosen variable with the highest absolute loading value is selected, and put in the ranking. In the non-iterative version this is made only once; in the iterative version, PCs are calculated again every time a variable is chosen and removed from the dataset.
Analysis of the dataset. The B2 method consists in a sequential analysis of all the Principal Components (PC), starting from the last one (the less significant). For each PC, the first not already chosen variable with the highest absolute loading value is selected, and put in the ranking. In the non-iterative version this is made only once; in the iterative version, PCs are calculated again every time a variable is chosen and removed from the dataset.
Clustering algorithms
Clustering is one of the best known algorithm in machine learning domain, named as an unsupervised learning algorithm. Clustering groups similar data points on a scatter plot for data visualization, prototyping, sampling, and segmentation. Clusters are the distinct groups that emerge from the segmentation process.
Clustering can lead to a single grouping/cluster or multiple clusters and can identify (previously unknown) groups in the data. The significance of clustering algorithm is to divide the large volume of data into smaller groups of data when there is no class labels available to process the datasets.
Each cluster contains a set of data points where clustering algorithm mainly used to classify and group each data point into a particular cluster. Besides, the data points within the same cluster should have similar properties, while data points in the difierent cluster should have highly dissimilar properties and/or eatures.
Many clustering algorithms for analyzing healthcare data sets have been introduced in the existing research works. Here we are discussing mainly clustering algorithms that are widely used in machine learning: K-Means algorithm, clustering by local gravitation, k-medoids clustering,k-means++
clustering.
Clustering can lead to a single grouping/cluster or multiple clusters and can identify (previously unknown) groups in the data. The significance of clustering algorithm is to divide the large volume of data into smaller groups of data when there is no class labels available to process the datasets.
Each cluster contains a set of data points where clustering algorithm mainly used to classify and group each data point into a particular cluster. Besides, the data points within the same cluster should have similar properties, while data points in the difierent cluster should have highly dissimilar properties and/or eatures.
Many clustering algorithms for analyzing healthcare data sets have been introduced in the existing research works. Here we are discussing mainly clustering algorithms that are widely used in machine learning: K-Means algorithm, clustering by local gravitation, k-medoids clustering,k-means++
clustering.
The k-means algorithm is one of the most popular clustering algorithms. It classifies the
dataset by dividing the samples into difierent clusters of equal variances. The number of
clusters must be specified in this algorithm. It is fast with fewer computations required,
with the linear complexity of O(n).
k-Means is one of the most popular "clustering"algorithms. K-means stores k centroids
that it uses to define clusters. A point is considered to be in a particular cluster if it is
closer to that cluster's centroid than any other centroid.
dataset by dividing the samples into difierent clusters of equal variances. The number of
clusters must be specified in this algorithm. It is fast with fewer computations required,
with the linear complexity of O(n).
k-Means is one of the most popular "clustering"algorithms. K-means stores k centroids
that it uses to define clusters. A point is considered to be in a particular cluster if it is
closer to that cluster's centroid than any other centroid.
The k-medoids problem is a clustering problem similar to k-means. Both the k-means and
k-medoids algorithms are partitional (breaking the dataset up into groups) and attempt
to minimize the distance between points labeled to be in a cluster and a point designated
as the center of that cluster. In contrast to the k-means algorithm, k-medoids chooses
actual data points as centers (medoids or exemplars), and thereby allows for greater
interpretability of the cluster centers than in k-means, where the center of a cluster is
not necessarily one of the input data points (it is the average between the points in
the cluster). Furthermore, k-medoids can be used with arbitrary dissimilarity measures,
whereas k-means generally requires Euclidean distance for eficient solutions.
Because kmedoids minimizes a sum of pairwise dissimilarities instead of a sum
of squared Euclidean distances, it is more robust to noise and outliers than k-means.
k-medoids algorithms are partitional (breaking the dataset up into groups) and attempt
to minimize the distance between points labeled to be in a cluster and a point designated
as the center of that cluster. In contrast to the k-means algorithm, k-medoids chooses
actual data points as centers (medoids or exemplars), and thereby allows for greater
interpretability of the cluster centers than in k-means, where the center of a cluster is
not necessarily one of the input data points (it is the average between the points in
the cluster). Furthermore, k-medoids can be used with arbitrary dissimilarity measures,
whereas k-means generally requires Euclidean distance for eficient solutions.
Because kmedoids minimizes a sum of pairwise dissimilarities instead of a sum
of squared Euclidean distances, it is more robust to noise and outliers than k-means.
The k-nearest neighbors algorithm (k-NN) is a non-parametric supervised learning method. It is used for classification and regression. In both cases, the input consists of the k closest training examples in a data set. The output depends on whether k-NN is used for classification or regression
The k-means problem is to find cluster centers that minimize the intra-class variance, i.e.
the sum of squared distances from each data point being clustered to its cluster center (the center that is closest to it). Although finding an exact solution to the k-means problem for arbitrary input is NP-hard, the standard approach to finding an approximate solution (often called Lloyd's algorithm or the k-means algorithm) is used widely and frequently finds reasonable solutions quickly.
the sum of squared distances from each data point being clustered to its cluster center (the center that is closest to it). Although finding an exact solution to the k-means problem for arbitrary input is NP-hard, the standard approach to finding an approximate solution (often called Lloyd's algorithm or the k-means algorithm) is used widely and frequently finds reasonable solutions quickly.
The objective of cluster analysis is to partition a set of data points into several groups based on a suitable distance measure. We use a model called local gravitation among data points. In this model, each data point is viewed as an object with mass, and associated with a local resultant force (LRF) generated by its neighbors
