"Recent advances in machine learning rely on collecting an enormous amount of data and learning immense models in a centralized cloud. However, the excessive storage and computational needs of centralized approaches, alongside regulatory challenges in sharing private data, put the utility of this paradigm in doubt. Collaborative machine learning is a recent alternative paradigm to tackle these issues by developing algorithms collaboratively without exchanging or centralizing the data. For example, different geographically distributed hospitals, each being in possession of limited patients’ data, may collaboratively develop predictive algorithms to improve diagnostics and treatment beyond what could be accomplished alone. Unlocking the full potential of collaborative learning strongly depends on the ability to encourage a large pool of individuals or corporations to share their private data and resources, while overcoming issues related to data and systems heterogeneity.

Despite recent progress on federated optimization, our understanding of some fundamental aspects of these methods, required for characterizing their performance guarantees, is still in its infancy. In this talk we will introduce a pluralistic heterogeneous distributed optimization framework for collaborative machine learning where the main idea is to integrate the personalization into the training and introduce a model shuffling idea to learn optimally from heterogeneous data sources. We also discuss novel stochastic communication-efficient distributed algorithms with provable convergence rates for adaptively exploiting underlying computational resources and overcoming the curse of data heterogeneity. Finally, we establish generalization bounds for the proposed algorithm and elaborate on different implications such as intensive compatibility and game theoretic implications."