Identify the parameters of the binomial expansion.

The given expansion is $\left(\frac{a}{2} + 2\right)^8$.
Here, $n=8$.
The number of terms in the expansion is $n+1 = 8+1 = 9$.

Since there are 9 terms, the middle term is the $\frac{9+1}{2} = 5^{th}$ term.
For the general term $T_{r+1}$, we have $r+1 = 5$, so $r=4$.

The general term in the expansion of $(x+y)^n$ is given by $T_{r+1} = \binom{n}{r} x^{n-r} y^r$.
In this case, $x = \frac{a}{2}$, $y = 2$, $n=8$, and $r=4$.

We are given that the middle term is 1120.

Take the fourth root of both sides:

The value of $a$ is $\boxed{\pm 2}$.