I recently created my own Google Apps for Business account for free (my company has less than 5 employees). I want to test Google Play Private Channel but it isn't possible with a free account, I can get a free trail for 30 days. I was wondering if I have to pay after the trial or I go back to my free account?
No. There's no longer a free edition of Google Apps. If you had signed up for a trial full account before December 6 2012, you could have cancelled the trial and gone back to the free edition. Since Google no longer offers a free edition, and December 6th is more than 30 days ago, you cannot revert to the free edition.
From the Google Support article: