In most collaborative filtering algorithms for the task of rating prediction, user ratings are often defined as numerical values or (nominal) categorical labels. Viewing rating scores in this way ignores the ordinality of user ratings. In practice, it is expected that the difference/scale of user preferences between any pair of adjacent rating scores should differ.  In our work, we propose a decomposed matrix factorization approach to modeling the ordinality of user ratings (i.e., modeling the differences between any pair of adjacent rating scores).  Besides,  our proposed method can be easily adapted to the task of top-N recommendation. Through extensive experiments on a variety of large-scale benchmark datasets, we empirically demonstrate the significant improvement of our proposed method over the state-of-the-art recommendation algorithms in terms of rating prediction accuracy and top-N ranking performance.