Innovating and Interpreting Neural Networks
Deep learning has recently achieved huge success in many applications, including natural language processing, computer vision and more. In these cases, deep learning can outperform or compete with humans. It is widely recognized that machine learning, especially deep learning, is a paradigm shift in many fields. However, there are still many challenges ahead. On one hand, over the past years, major efforts have been dedicated to architecture innovations in the field of neural networks, leading to many advanced models. Although deep learning is inspired by the computation of the neural system, current deep learning systems fall short of reflecting neural diversity. On the other hand, despite the fact that deep learning performs quite well in practice, it is difficult to explain its underlying mechanism and understand its behaviors. The success of deep learning is not well underpinned by the effective theory. Lacking interpretability has become a primary obstacle to the widespread translation and further development of deep learning techniques. In this project, we propose quadratic neurons to address the neural diversity problem in deep learning, where inner products (which are linear operations) are replaced with quadratic counterparts whose non-linearity enhances the expressive ability of the neuron. Further, we propose soft thresholding to replace ReLU activation for signal processing tasks. We will evaluate their feasibility in practical computer vision problems as well as medical imaging problems, thereby enriching machine learning armory. We will also develop interpretation methods for the inner working of neural networks and accountable theories for the success of deep networks.
