I've found no way to do this on the website or in the android app.
Is there any way to unfollow people on fitocracy and remove them from your friend list?
Go to the Profile page of person you want to unfollow. You will find a wrench icon called "Controls" right bellow the "Quests". Open the "Controls" by clicking on the little arrow next to it, then you will found the "unfollow" button.