From the figure:

$\theta = \alpha - \beta$

$\tan \theta = \tan (\alpha - \beta)$

$\tan \theta = \dfrac{\tan \alpha - \tan \beta}{1 + \tan \alpha \, \tan \beta}$

$\tan \theta = \dfrac{\dfrac{b}{x} - \dfrac{a}{x}}{1 + \dfrac{b}{x} \left( \dfrac{a}{x} \right)}$

$\tan \theta = \dfrac{\dfrac{b - a}{x}}{\dfrac{x^2 + ab}{x^2}}$

$\tan \theta = \dfrac{b - a}{\dfrac{x^2 + ab}{x}}$

$\tan \theta = \dfrac{(b - a)x}{x^2 + ab}$

$\theta = \arctan \dfrac{(b - a)x}{x^2 + ab}$

$\dfrac{d\theta}{dx} = \dfrac{\dfrac{(x^2 + ab)(b - a) - (b - a)x(2x)}{(x^2 + ab)^2}}{1 + \left[ \dfrac{(b - a)x}{x^2 + ab} \right]^2} = 0$

$\dfrac{(x^2 + ab)(b - a) - (b - a)x(2x)}{(x^2 + ab)^2} = 0$

$(x^2 + ab)(b - a) - (b - a)x(2x) = 0$

$(b - a) \, [ \, (x^2 + ab) - 2x^2 \, ] = 0$

$(x^2 + ab) - 2x^2 = 0$

$ab - x^2 = 0$

$ab = x^2$

$x = \sqrt{ab}$ *answer*