I believe that the only things that affect your friend count are: if someone friends/unfriends you, someone activates/deactivates their account, or if someone deletes their account permanently. I don't think there are any privacy settings that would prevent you from seeing which friends you have.
Deleting a profile:
You'll notice that the query
delete facebook has been constantly on the rise, but a few months ago, there was a sharp fluctuation; this was probably because of Quit Facebook Day where people permanently deleted their Facebook accounts. It could just be that people are fed up with the privacy issues and are deleting their profiles.
Deactivating a profile:
I know that I've had friends that find themselves on Facebook way too much and deactivate their account in a attempt to be more productive. This usually fails though, as they reactivate their accounts shortly after.
As far as unfriending you, you seem like a pretty decent guy, so that's probably not the reason the numbers are changing : ) However, you could try to use one of these methods to see which friends are being removed from the list. Once you find someone that is no longer on your friend list, you can try to access their profile. If you can access it, that means they unfriended you. If you can't, they either unfriended and blocked you, deactivated their account, or deleted their account.