Sangwha Lee from Krea.ai presented an overview of the training process behind their image foundation model, Create 2, focusing primarily on research aspects. The team recently open-sourced a medium variant of this model, emphasizing stylistic diversity and faster generation speeds compared to other production-grade models like ChatGPT-2 or Nano Banana Pro. While those models prioritize reliability and consistency, often at the cost of diversity, Create 2 aims to support creative exploration by enabling quicker iterations and more varied outputs, which is particularly useful for creative studios still exploring visual ideas.
The core of their model training involves diffusion models, specifically latent diffusion models that operate in a compressed latent space rather than raw pixels. This approach improves efficiency since transformers, commonly used in diffusion models, have quadratic time complexity with respect to input size. Sangwha stressed that data quality and curation are paramount, often more critical than architectural changes. Their focus was on maintaining stylistic diversity by avoiding over-filtering based on conventional image quality metrics, which might exclude aesthetically valuable but unconventional images.
Data curation involved several sophisticated techniques, including deduplication using hash-based and embedding-based methods, filtering out AI-generated images to avoid model shortcuts, and improving caption quality through OCR and vision-language models. They also used sparse autoencoders for unsupervised tagging to identify and filter undesirable data features like watermarks or borders. Additionally, they incorporated world knowledge by ensuring important concepts from Wikipedia were well represented in the training data, enhancing the model’s understanding of relevant subjects.
The training pipeline was inspired by large language model (LLM) methodologies, progressing from low to high resolution to first learn semantics and then finer details. After pre-training, the model undergoes mid-training with curated data, supervised fine-tuning, preference optimization based on human feedback, and reinforcement learning to improve specific capabilities like text rendering and anatomy. A unique step involves training a prompt expander—a small LLM that converts user prompts into detailed prompts better aligned with the model’s training distribution, improving output quality.
Looking ahead, Sangwha expressed interest in simplifying the training stack by potentially eliminating components like VAEs and text encoders to train a single clean transformer. He also highlighted exciting directions such as conditioning image generation on bounding boxes and scene graphs, enabled by advances in vision-language models. These approaches could provide richer textual descriptions and control over generated images. Finally, he invited interested researchers to connect with Krea.ai, emphasizing the importance of infrastructure, data, and efficient iteration in advancing generative image models.
Useful Links
