Suppose R is the user-rating matrix and you decompose R as R = P.Q<sup>T</sup> then use NN for improving P and Q<sup>T</sup>.
Suppose R is the user-rating matrix and you decompose R as R = P.QT then use NN for improving P and QT.