I think the question to ask is whether or not your pull request was merged or not?
If none of your pull requests have been merged and they are still just sitting there closed then I would do these following steps:
Delete your pull request
Delete the forked repository
From experience (similar situation as you) I believe the above steps worked, and I was removed as a contributor to the repository, but this GitHub account has since been deleted entirely so I don't have any way to go back and verify this works.
But you don't have anything to lose, if you try those steps, since you've already decided you want to try to erase this history.
If your pull request was merged, I believe the only way to remove the "repositories contributed to" thing, is to delete your account, unfortunately.