The rest of the wall for hundreds of metres in both directions does not have plant life growing in the gaps and the rest of the wall doesn't have lumps of rock coming loose. Plant roots might not look all that strong but the do have the effect you can see above. Imagine what effect tree roots would have on this wall. Imagine what effect the tree roots were having on Rock Walk.
Which basically was why the trees had to go.