calque
calque

Workshops

Deep learning in materials science: interpretation, generalization, and the risk of overfitting

September 23, 2019 to September 25, 2019
Location : Kartause Ittingen (Switzerland) (https://www.kartause.ch/en)
   Visa requirements

Organisers

  • Luca M. Ghiringhelli (Fritz Haber Institute of the Max Planck Society (FHI), Berlin , Germany)
  • Angelo Ziletti (Fritz Haber Institute of the Max Planck Society (FHI), Berlin , Germany)
  • Matthias Scheffler (Fritz Haber Institute of the Max Planck Society (FHI), Berlin, Germany)

Supports

   CECAM

   Psi-k



Description

Thanks to projects like the NOMAD Laboratory [1], Materials Project [2], AFLOW [3] and available public data at the institutes, a large amount of high-quality computational materials-science data are now available to the scientific community. This represents a unique opportunity for uncovering the underlying patterns and structure in the data and thus enable the development of structure-property relationships in materials science data.
Big-data driven materials science - the so-called fourth paradigm in materials science - recognizes that it may not be possible, or not appropriate, to describe many properties of functional materials by a single, physically founded model [C. Draxl and M. Scheffler,
NOMAD: The FAIR Concept for Big-Data-Driven Materials Science. Invited Review for MRS Bulletin (to appear September 2018); https://arxiv.org/abs/1805.05039]. The reason is that such properties are determined by several multi-level, intricate theoretical concepts. Thus, insight is obtained by searching for structure and patterns in the data, which arise from functional relationships among different processes and functions. Finding these relationships could lead, for example, to the discovery of efficient catalysts for methane formation, good thermal-barrier coatings, shape memory alloys, certain quantum-states of matter (e.g. topological insulators), thermoelectric materials or new perovskites for energy harvesting.

Given that deep learning is revolutionizing many areas (e.g. speech recognition, image classification, natural-language generation), it is reasonable to believe that deep learning could play a significant role in the development of data-driven materials science. Indeed, very recently deep learning has been successfully applied to a large variety of materials science problems[4-9]. In general, the origin of the deep learning success can be traced back to its ability to automatically construct representations which capture important aspects of the data, while discarding inessential details. However, it is challenging to specify further this high-level perspective and provide a detailed explanation on how the model arrives at its prediction. Thus, usually we are left with a black-box model, which from a scientific standpoint is not fully satisfactory, and that we would wish to open and understand.

Even though numerous methods have been recently proposed to interpret deep learning models (e.g., [9-14]), somewhat surprisingly, interpretability in deep learning - and more generally in machine learning - is far from being a consensual concept, with diverse and sometimes contrasting motivations for it [15]. If we turn the attention to machine learning for materials science, the situation is even less clear.

Reasonable candidate properties of interpretable models could be model transparency (i.e. how does the model work?) and post hoc explanations (i.e., what else can the model tell me?).

In the materials science/physics domain, transparency could be demonstrated by justifying the model from first principles, for example by quantum-mechanical calculations or an analytical model valid for some regime, and or by molecular dynamics and statistical-mechanics analysis. In addition, post hoc explanations could be used to show that the deep-learning model is predictive well beyond training data; these new data points could be suggested by inspection of the analytical model or physical considerations.

From an intuitive perspective, interpretability is - at least in natural sciences - intrinsically linked with generalizability, although this link is yet to be defined.

[1] https://nomad-coe.eu
[2] http://materialsproject.org
[3] http://aflowlib.org
[4] Ghasemi SA, Hofstetter A, Saha S, Goedecker S. Interatomic potentials for ionic systems with density functional accuracy based on charge densities obtained by a neural network, Phys. Rev. B 92, 045131 (2015)
[5] Morawietz T, Singraber A, Dellago C, Behler J. How van der Waals interactions determine the unique properties of water, PNAS 113 (30) 8368-8373 (2016)
[6] Schütt KT, Arbabzadah F, Chmiela S, Müller KR, Tkatchenko A. Quantum-chemical insights from deep tensor neural networks, Nat. Comm. 8, 13890 (2017)
[7] Ryan K, Lengyel J, Shatruk M. Crystal Structure Prediction via Deep Learning, JACS, in press (2018)
[8] T. Xie and J. C. Grossman, Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties, Phys. Rev. Lett. 120, 145301 (2018)
[9] Ziletti A, Kumar D, Scheffler M, Ghiringhelli LM. Insightful classification of crystal structures using deep learning, Nat. Comm, in press, arXiv:1709.02298 (2018)

[10] Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation, PLOS ONE 7 (10), e0130140 (2015)
[11] Ribeiro MT, Singh S, Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier, in "Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining", ACM, pp. 1135-1144 (2016). arXiv:1602.04938
[12] S. Lundberg and S. Lee, A Unified Approach to Interpreting Model Predictions, in "Advances in Neural Information Processing Systems 2017", pp. 4765-4774. arXiv: 1705.07874 (2017)
[13] Kumar D, Wong A, Taylor GW. Explaining the Unexplained: A CLass-Enhanced Attentive Response (CLEAR) Approach to Understanding Deep Neural Networks, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1686 (2017)
[14] Montavon G, Samek W, Müller KR. Methods for interpreting and understanding deep neural networks, Digital Signal Processing (73), 1 (2018)
[15] Z. C. Lipton, The Mythos of Model Interpretability, ICML Workshop on Human Interpretability in Machine Learning (WHI 2016), arXiv:1606.03490 (2016)

References

[1] https://nomad-coe.eu
[2] http://materialsproject.org
[3] http://aflowlib.org
[4] Ghasemi SA, Hofstetter A, Saha S, Goedecker S. Interatomic potentials for ionic systems with density functional accuracy based on charge densities obtained by a neural network, Phys. Rev. B 92, 045131 (2015)
[5] Morawietz T, Singraber A, Dellago C, Behler J. How van der Waals interactions determine the unique properties of water, PNAS 113 (30) 8368-8373 (2016)
[6] Schütt KT, Arbabzadah F, Chmiela S, Müller KR, Tkatchenko A. Quantum-chemical insights from deep tensor neural networks, Nat. Comm. 8, 13890 (2017)
[7] Ryan K, Lengyel J, Shatruk M. Crystal Structure Prediction via Deep Learning, JACS, in press (2018)
[8] T. Xie and J. C. Grossman, Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties, Phys. Rev. Lett. 120, 145301 (2018)
[9] Ziletti A, Kumar D, Scheffler M, Ghiringhelli LM. Insightful classification of crystal structures using deep learning, Nat. Comm, in press, arXiv:1709.02298 (2018)

[10] Bach S, Binder A, Montavon G, Klauschen F, Müller KR, Samek W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation, PLOS ONE 7 (10), e0130140 (2015)
[11] Ribeiro MT, Singh S, Guestrin C. “Why Should I Trust You?” Explaining the Predictions of Any Classifier, in "Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining", ACM, pp. 1135-1144 (2016). arXiv:1602.04938
[12] S. Lundberg and S. Lee, A Unified Approach to Interpreting Model Predictions, in "Advances in Neural Information Processing Systems 2017", pp. 4765-4774. arXiv: 1705.07874 (2017)
[13] Kumar D, Wong A, Taylor GW. Explaining the Unexplained: A CLass-Enhanced Attentive Response (CLEAR) Approach to Understanding Deep Neural Networks, IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1686 (2017)
[14] Montavon G, Samek W, Müller KR. Methods for interpreting and understanding deep neural networks, Digital Signal Processing (73), 1 (2018)
[15] Z. C. Lipton, The Mythos of Model Interpretability, ICML Workshop on Human Interpretability in Machine Learning (WHI 2016), arXiv:1606.03490 (2016)