Thursday, 3rd of November 2022, 12:00 – 1:00

Causes and effects of unanticipated numerical deviations in deep learning frameworks


Alex Schlögl - researcher at SEC


For a fixed training model and fixed input data, inference results are not consistent across hardware configurations, and sometimes not even deterministic on the same hardware configuration. We performed a deep dive into a typical machine learning inference stack and identified algorithm selection, floating point accuracy, aggregation order, data parallelism, and task parallelism as causes for numerical deviations. I will present existential evidence for our identified root causes, highlighting the complex interaction between TensorFlow, CUDA, and the Eigen linear algebra library. I will also show how these factors combine to yield different results on a large set of different hardware configurations. Finally, I will briefly discuss the implications these deviations can have on forensics, machine learning security, and applications of machine learning.

Nach oben scrollen