Why did you use sine² And why can't you use Pythagoras theorem

You can use the Pythagorean theorem indirectly. If you imagine a right-angled triangle where $x$ is one of the acute angles, and $\cos x = \frac{\text{adjacent}}{\text{hypotenuse}} = \sqrt{\frac{a}{b}}$, you could label the adjacent side as $\sqrt{a}$ and the hypotenuse as $\sqrt{b}$. Then, using the Pythagorean theorem, the opposite side would be $\sqrt{(\sqrt{b})^2 - (\sqrt{a})^2} = \sqrt{b-a}$.
From this, $\sin x = \frac{\text{opposite}}{\text{hypotenuse}} = \frac{\sqrt{b-a}}{\sqrt{b}} = \sqrt{\frac{b-a}{b}}$.
This is the same result obtained using the identity. The Pythagorean identity is essentially a direct application of the Pythagorean theorem to the coordinates on a unit circle or the sides of a right triangle.