M=(I_n-\frac{1}{n}11^T)

<math\; xmlns="http://www.w3.org/1998/Math/MathML"\; display="flex"\; mathsize="1em">\; <mrow>\; <msup>\; <mi><br></mi>\; </msup>\; </mrow>\; </math>

<math\; xmlns="http://www.w3.org/1998/Math/MathML"\; display="flex"\; mathsize="1em"><mrow><msup><mi><math\; xmlns="http://www.w3.org/1998/Math/MathML"\; display="flex"\; mathsize="1em"><mrow><mi><br></mi></mrow></math></mi></msup></mrow></math>

The correct answer is 1.

Here is the step-by-step explanation of how to find the maximum value of the quadratic form.

Step 1: Understand the Given Matrix and Set
We are given a matrix $M$ defined as:
$$M=I_n-\frac{1}{n}11^T$$
where $I_n$ is the $n\times n$ identity matrix, and $1={(1,1,\dots ,1)}^T$ is a column vector of all ones of dimension $n$.
We want to maximize the expression $x^{T}Mx$ over the set of unit vectors:
$$S=\{x\in R^n|x^{T}x=1\}$$

Step 2: Relate to Eigenvalues (Rayleigh Quotient)
For any symmetric matrix $M$, the maximum value of $x^{T}Mx$ subject to the constraint $x^{T}x=1$ is equal to the largest eigenvalue (${\lambda }_{\max }$) of the matrix $M$:
$$\underset{x^{T}x=1}{\max }{x}^{T}Mx={\lambda }_{\max }\left(M\right)$$

Step 3: Analyze the Matrix Properties
Let us check the properties of $M$.
First, $M$ is symmetric because:
$$M^T={(I_n-\frac{1}{n}11^T)}^T=I_n^T-\frac{1}{n}(11^T{)}^T=I_n-\frac{1}{n}11^T=M$$
Second, $M$ is an idempotent matrix (a projection matrix):
$$M^2=(I_n-\frac{1}{n}11^T)(I_n-\frac{1}{n}11^T)$$
Expanding this product gives:
$$M^2=I_n-\frac{2}{n}11^T+\frac{1}{n^2}1\left(1^{T}1\right)1^T$$
Since $1^{T}1=n$ (the dot product of the ones vector with itself is $n$), we get:
$$M^2=I_n-\frac{2}{n}11^T+\frac{n}{n^2}11^T=I_n-\frac{1}{n}11^T=M$$

Step 4: Find the Eigenvalues of $M$
Since $M$ is an idempotent projection matrix, its eigenvalues can only be 0 or 1.
Let us find the eigenvectors corresponding to these eigenvalues:
1. If we multiply $M$ by the vector $1$:
$$M1=(I_n-\frac{1}{n}11^T)1=1-\frac{1}{n}1\left(1^{T}1\right)=1-\frac{n}{n}1=0$$
Thus, $\lambda =0$ is an eigenvalue of $M$ corresponding to the eigenvector $1$ (with multiplicity 1).

2. For any vector $v$ that is orthogonal to $1$ (meaning $1^{T}v=0$):
$$Mv=(I_n-\frac{1}{n}11^T)v=v-\frac{1}{n}1\left(1^{T}v\right)=v-0=v$$
Thus, $\lambda =1$ is an eigenvalue of $M$ with multiplicity $n-1$ (corresponding to the subspace orthogonal to $1$).

Conclusion
Since the eigenvalues of $M$ are 0 and 1, the maximum eigenvalue of $M$ is 1. Therefore, the maximum value of $x^{T}Mx$ subject to $x^{T}x=1$ is 1.