In many of machine learning problems, it is essential to use not only the training data, but also a priori knowledge about how the world is constrained. In many cases, such knowledge is given in the forms of constraints on differential data or more specifically partial differential equations (PDEs). Neural networks with capabilities to learn differential data can take advantage of such knowledge and easily incorporate such constraints into the learning of training value data. In this paper, we report a structure, an algorithm, and results of experiments on neural networks learing differential data.