Logically looking at this, three people have gone from working for the family "Firm" representing HM, The Queen. Andrew has effectively been retired from performing any kind of public duties and most (if not all) of his private patronages have dumped him.
Should the Sussexes return like the prodigal son to the fold of the family "Firm" (which I do believe is very unlikely at all), there will always be the niggling thought in the back of people's minds that if they've walked away before, they very well could walk away again should they experience displeasure in how things are.
There is a statement that says "You can't go home again. You can't truly go back to a place you once lived because so much will have changed since you left that it is not the same place anymore." and I think this applies to this situation.