Karsten Kreis: Visual Generative AI with Diffusion Models – From Static Pixels to Video, 3D and 4D Synthesis

We are currently witnessing a generative AI revolution, not only in language modeling, but also in the visual domain. While only a decade ago the generation of detailed imagery seemed impossible, we can now routinely synthesize detailed and highly expressive images, videos, and even 3D and dynamic 4D content. In this talk, I will present a journey through the field of generative AI in the visual domain and explain some of the most important methods and techniques. 

I will review the foundations of diffusion models, which have led to major breakthroughs in large-scale image generation and now power most modern deep generative learning systems in the visual domain. I will also mention some of our works on advanced diffusion processes and accelerated diffusion model sampling. Next, I will explain how diffusion models are used not only for image generation but can be extended to video generation, a task where we have seen rapid progress very recently.

Further, I will discuss how text-to-image diffusion models are now widely utilized for static 3D object synthesis through score distillation sampling, an approach that optimizes individual 3D objects based on feedback from a pre-trained image diffusion model. Finally, I will show our work on combining score distillation sampling with video diffusion models for the generation of moving 3D objects, thereby enabling 4D dynamic content creation with diffusion models.

Karsten Kreis is a Senior Research Scientist in the fundamental generative AI research team of NVIDIA Research. Karsten is interested both in fundamental generative AI algorithm development and in applying deep generative models in areas such as computer vision, graphics and digital artistry as well as to problems in the natural sciences. Before joining NVIDIA, Karsten worked on deep generative learning at D-Wave Systems and he co-founded Variational AI, a startup leveraging generative modeling for drug discovery. Karsten is trained as a physicist and did his master’s in quantum information theory and his Ph.D. in computational and statistical physics, developing multiscale molecular dynamics simulation methods. See his website for more details.