Despite mimicking some of the LaTeX syntax, Google Document Equation Editor is still WYSIWYG: changes to an equation happen immediately in response to key presses, there is no compilation stage. In particular, there are no { }
for grouping: you control the placement of characters visually. Think of entering "\frac " as equivalent to clicking a button for "fraction pattern" in the toolbar.
To enter a fraction of the form a/b, type
\frac aTabbTab
The fraction structure appears first, with the cursor set in the numerator. After entering "a" there,pressing Tab
moves the cursor to the denominator. The second Tab moves it out of the denominator, so you can continue with the formula.
Instead of Tab, the right arrow key may be used.