In the OpenAI DevDay 2024 presentation, Atty Eleti and Michelle Pokrass introduced structured outputs, a feature designed to enhance the reliability of applications using large language models by ensuring outputs adhere to specified JSON schemas. They explained how this innovation improves upon previous methods by eliminating issues with invalid JSON and hallucinations, ultimately streamlining the development process for developers.
In the OpenAI DevDay 2024 presentation, Atty Eleti and Michelle Pokrass introduced the concept of structured outputs, a feature launched in August 2023 that enhances the reliability of applications using large language models (LLMs). They highlighted the evolution of OpenAI’s models from GPT-3 to GPT-4, emphasizing the increasing complexity and capabilities of these models. Developers have leveraged these advancements to create various applications, but the challenge of obtaining reliable outputs, particularly in structured formats like JSON, has persisted. The introduction of structured outputs aims to address these issues by ensuring that model-generated outputs strictly adhere to the specified JSON schemas.
The presenters explained that structured outputs provide a more robust solution compared to previous methods, such as function calling and JSON mode. While function calling allowed developers to define tools for the model to use, it sometimes resulted in invalid JSON outputs. JSON mode improved the situation by ensuring valid JSON, but it still did not guarantee the correct types or prevent hallucinations. Structured outputs eliminate these problems by constraining the model’s outputs to match the provided schemas, thus enhancing the reliability of AI applications.
Michelle detailed how structured outputs can be utilized in two modes: function calling and response format parameters. In function calling, developers can supply a JSON schema that the model must follow, ensuring that the generated outputs are valid and correctly formatted. The response format parameter allows developers to specify how the model should respond directly to users, ensuring that the outputs are concise and fit within the application’s constraints. The presenters demonstrated these features through practical examples, showcasing how structured outputs can streamline the development process and reduce errors.
The engineering behind structured outputs involves constrained decoding, which ensures that the model adheres to the specified schemas during inference. This process includes token masking, where only valid tokens are allowed based on the current state of the output being generated. The presenters explained that this approach allows for efficient inference while maintaining the flexibility needed for complex JSON schemas, including recursive structures. They also discussed the importance of building an index for fast lookups during inference, which helps maintain performance even at scale.
Finally, the presentation highlighted the research efforts that improved the models’ ability to follow formats accurately. By training the models on complex schemas and ensuring they understand the semantic meaning behind different fields, OpenAI has significantly enhanced the accuracy of outputs. The presenters concluded by emphasizing their commitment to providing developers with the tools needed to build reliable AI applications, expressing excitement about the potential innovations that structured outputs will enable in the future.