Streaming sound...
The streaming sound caused frame-skipping which made my paddle jump around. Also, some detection of whether the mouse is on-screen would prevent the frustration of the paddle suddenly becoming unresponsive.
Really cool idea with the stick figures though, nice spin on a classic, but the execution needs help.