The fastest way to do it if you want to remove a lot of labels:
Position the cursor over the card and then press l and then the number of the label. You find the number of the label when you open a card and press on the label field.
Click on the label you wish to remove. This takes you to the labels menu. There should be a small "check" on the label that is showing. Click on the "check" to deselect it. This should make the label disappear from the card.