He may have been stripped of his royal status, but for some, the late Queen’s second son should not be vilified any further ...