<p>In centroid-based clustering, clusters are represented by some central vector which may or may not be a member of the dataset. In practice, the number of clusters is fixed to $k$ and the goal is to solve some sort of optimization problem.</p>
<p>The similarity of two clusters is defined as the similarity of their centroids.</p>
<p>This problem is computationally difficult so there are efficient heuristic algorithms that are commonly employed. These usually converge quickly to a local optimum.</p>
<h2>K-means clustering</h2>
<p>This aims to partition $n$ observations into $k$ clusters in which each observation belongs to the cluster with the nearest mean which serves as the centroid of the cluster.</p>
<p>This technique results in partitioning the data space into Voronoi cells.</p>
<h3>Description</h3>
<p>Given a set of observations $x$, k-means clustering aims to partition the $n$ observations into $k$ sets $S$ so as to minimize the within-cluster sum of squares (i.e. variance). More formally, the objective is to find
where $\mu_i$ is the mean of points in $S_i$. This is equivalent to minimizing the pairwise squared deviations of points in the same cluster
$$
argmin<em>s{\sum</em>{i = 1}^k{\frac{1}{2|S<em>i|}\sum</em>{x, y \in S_i}{||x-y||^2}}}
$$</p>
<h3>Algorithm</h3>
<p>Given an initial set of $k$ means, the algorithm proceeds by alternating between two steps.</p>
<p><strong>Assignment step</strong>: Assign each observation to the cluster whose mean has the least squared euclidean distance.</p>
<ul>
<li>Intuitively this is finding the nearest mean</li>
<li>Mathematically this means partitioning the observations according to the Voronoi diagram generated by the means</li>
</ul>
<p><strong>Update Step</strong>: Calculate the new means to be the centroids of the observations in the new clusters</p>
<p>The algorithm is known to have converged when assignments no longer change. There is no guarantee that the optimum is found using this algorithm. </p>
<p>The result depends on the initial clusters. It is common to run this multiple times with different starting conditions.</p>
<p>Using a different distance function other than the squared Euclidean distance may stop the algorithm from converging.</p>
<h3>Initialization methods</h3>
<p>Commonly used initialization methods are Forgy and Random Partition.</p>
<p><strong>Forgy Method</strong>: This method randomly chooses $k$ observations from the data set and uses these are the initial means</p>
<p>This method is known to spread the initial means out</p>
<p><strong>Random Partition Method</strong>: This method first randomly assigns a cluster to each observation and then proceeds to the update step. </p>
<p>This method is known to place most of the means close to the center of the dataset.</p>
