You got pretty close with your idea about ISBLANK.
Use this as the Custom formula applying to Columns B to D
The formula evaluates two ranges:
1) cell A = whether it is NOT empty
2) cell B or C or D = whether they are blank (that's the impact of the absolute column reference).
The criterion are joined with AND. So the net effect is as if to say:
If cell A is NOT empty, then apply the formatting to the cells in Column B, C, and D UNLESS the respective cell B, C or D is not empty in which case don't apply the formatting to that non-empty cell.
This is the screenshot of the indicative outcome.
Absolute Vs Relatives references in critera: Zig Mandel and tehhowch
Formatting Rule applying to a column: Ed Nelson