I've always thought visiting graves to be morbid and pointless, and I think it is an odd thing to do. I believe that people live on in their friends' and loved ones' hearts and memories, not in the immediate vicinity of the box where their mortal remains are buried. You certainly don't have to visit the place those remains are buried to remember the person who once occupied that body. I know some people believe it is a good thing to do though, and important. I don't criticise anyone for that; if it feels right for them, it's the right thing to do. I'm just providing the point of view of someone who doesn't see any need whatsoever for Charles to ever have visited Diana's grave.
Charles can't possibly forget Diana; for one thing the press won't let him. For another, he has their two sons as a constant reminder. But apart from that she was a part of his life for more than 15 years. He lived with her and worked with her, and had children with her, and there will be numerous things in his life that will remind him of Diana.
I'm sure he has many happy memories of Diana, and that it is likely those are the ones he recalls. I doubt he wants to be reminded of the tragic events that preceded her burial - the phone call in the night, dellivering the heartbreaking news to their sons, the flight to Paris, seeing her dead body lying in the hospital, searching for her earring, taking her home, dealing with their sons' grief, the lead-up to the funeral, the funeral itself, her brother's speech at the funeral, the public's applause, taking her to her burial place, placing her in the ground, and leaving her there.
And even if he did want to visit her grave, Diana is buried on "hostile" ground. If he goes, his presence will be noticed and commented upon in the press. No, it's best he stays away and remembers her in his own way.