The increasing popularity of smartphone and wrist-mounted devices creates more potential for sensing driver actions and guiding phone usage. Based on the inertial sensors of smartphones and wrist-mounted devices, we proposed a model to distinguish a driver's phone use from a passenger's with 98.9% accuracy in our tests, which can facilitate many traffic safety applications. To further track unsafe driving behaviors, we designed a method to detect whether a driver’s hand is on or off steering wheel, and archived 99.8% accuracy in our tests. Furthermore, instead of completely prohibiting phone usage while driving, we proposed an architecture to channel a driver’s phone usage into relative safer periods, and implemented the system in the context of waiting for traffic lights. This requires predicting when a sufficiently long safe period for phone interaction exists. Simulations and experimental evaluation show that the system can achieve a low prediction error and its convergence and prediction accuracy increase proportionally with the amount of available crowd-sourced data.