For largest rectangle
$x = \dfrac{a}{\sqrt{2}}$ and $y = \dfrac{b}{\sqrt{2}}$
 

 

$A_{max} = 4xy$

$A_{max} = 4 \left( \dfrac{5}{\sqrt{2}} \right) \left( \dfrac{4}{\sqrt{2}} \right)$

$A_{max} = 40 ~ \text{unit}^2$

$\dfrac{x^2}{25} + \dfrac{y^2}{16} = 1$

$y^2 = 16\left( 1 - \dfrac{x^2}{25} \right)$

$y^2 = 16\left( \dfrac{25 - x^2}{25} \right)$

$y^2 = \frac{16}{25}(25 - x^2)$

$y = \frac{4}{5} \sqrt{25 - x^2}$
 

 

Area of rectangle inscribed in the ellipse
$A = 4xy$

$A = 4x \left( \frac{4}{5} \sqrt{25 - x^2} \right)$

$A = \frac{16}{5}x \sqrt{25 - x^2}$
 

For the largest rectangle
$\dfrac{dA}{dx} = \frac{16}{5} \left[ x \cdot \dfrac{-2x}{2\sqrt{25 - x^2}} + \sqrt{25 - x^2} \cdot 1 \right] = 0$

$x \cdot \dfrac{-2x}{2\sqrt{25 - x^2}} + \sqrt{25 - x^2} \cdot 1 = 0$

$\dfrac{-x^2}{\sqrt{25 - x^2}} + \sqrt{25 - x^2} = 0$

$\sqrt{25 - x^2} = \dfrac{x^2}{\sqrt{25 - x^2}}$

$25 - x^2 = x^2$

$2x^2 = 25$

$x = \frac{5}{\sqrt{2}}$
 

Area of the largest rectangle
$A_{max} = \frac{16}{5} \cdot \frac{5}{\sqrt{2}} \cdot \sqrt{25 - \frac{25}{2}}$

$A_{max} = 40 ~ \text{unit}^2$