Existing works on decision making system mainly focus on improving performance, but they rarely provide explanation how the decision is made and why it is relevant to users. In this work, we believe that equipping the decision-making system with knowledge graph provides an alternative to make explainable decision accompanied by an explicit reasoning process over the graph. Particularly, we consider an explainable recommendation problem over knowledge graph. The goal is to learn to reason over the graph to find possible paths from users to items for recommendation, and the reasoning paths can be naturally regarded as one interpretation for the decision. The work studies how to make better and explainable decisions through explicit reasoning process over the graph. The proposed methods are expected to be applicable to a wide scope of graph reasoning tasks, such as recommendation system, social network analysis, citation network analysis and medical graph reasoning for e-health research, etc.