Analysis of medical images is essential in modern medicine. With the increasing amount of patient data, new challenges and opportunities arise for different phases of the clinical routine, such as diagnosis, treatment and monitoring. The InnerEye research project focuses on the automatic analysis of patients’ medical images. It uses state of the art machine learning techniques for the: ΓÇó automatic delineation and measurement of healthy anatomy and anomalies; ΓÇó robust registration for monitoring disease progression; ΓÇó semantic navigation and visualization for improved clinical workflow; ΓÇó development of natural user interfaces for medical practitioners. Our mission is to advance the state of the art in machine learning and marry it with medical expertise, with application in computer-aided diagnosis, personalized medicine and efficient data management. Some of this technology is now incorporated within the Microsoft Amalga Unified Intelligence System.