Controlling Output Format | Ucode

Appearance

On this page

Controlling Output Format ​

When developing LLM applications, we typically don't present the model's generated answers directly to users. Instead, the model's output is often treated as an intermediate step that requires further processing.

For example, in creating an intelligent travel planning assistant, a user may input their travel preferences, and the model generates a travel plan with recommendations for destinations, routes, and transportation. The output is then formatted, and based on the parsed data, relevant APIs for flight bookings, hotel reservations, and weather forecasts are called to further refine the travel plan.

To format the model's text output, LangChain provides a series of output parsers to meet various parsing needs. A complete LLM chain generally consists of three factors: prompt input, LLM call, and output parsing.

format1.webp

In this lesson, we will first explore how LangChain integrates output parsers into the LLM chain execution flow, and then delve into the workings and usage of various common output parsers. This will help in selecting the most suitable output parser for specific requirements in actual development work.

Output Parsers in LangChain ​

Although output parsers appear after the LLM call when describing an LLM chain, in LangChain’s design, the output parser already plays a role in the construction of the prompt.

The working principle of an output parser is simple: it reserves a placeholder variable in the prompt template for the output format, which the output parser is responsible for filling. After the LLM returns a text response according to the output requirements, the output parser parses it into the expected data structure.

format2.webp

Each output parser implements two methods:

  • get_format_instructions: Fills in the placeholder variable in the prompt template with the output format, constraining the text format returned by the template.
  • parse: Accepts the text response from the LLM and parses it into a specified data structure.

Let’s use the CommaSeparatedListOutputParser in LangChain to demonstrate what an LLM chain looks like with an output parser and how it affects the result.

CommaSeparatedListOutputParser ​

The CommaSeparatedListOutputParser is provided by LangChain for returning output in array format.

python
from langchain.output_parsers import CommaSeparatedListOutputParser
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

# 1. Instantiate a CommaSeparatedListOutputParser object
output_parser = CommaSeparatedListOutputParser()

format_instructions = output_parser.get_format_instructions()

# 2. Create a prompt template and fill in the output_parser.get_format_instructions() in the final prompt
prompt = PromptTemplate(
    template="List five {subject}.\n{format_instructions}",
    input_variables=["subject"],
    partial_variables={"format_instructions": format_instructions},
)

# 3. Create an LLM instance
model = ChatOpenAI(temperature=0)

# 4. Build the chain
chain = prompt | model | output_parser

# Invoke chain and observe the LLM chain return result
print(chain.invoke({"subject": "colors of the rainbow"}))
#> List['red', 'orange', 'yellow', 'green', 'blue']

Here, the chain returns a Python array, which can be used directly for subsequent logic processing.

The CommaSeparatedListOutputParser works by generating format instructions:

python
class CommaSeparatedListOutputParser(ListOutputParser):
    ...
    def get_format_instructions(self) -> str:
        return (
            "Your response should be a list of comma separated values, "
            "eg: `foo, bar, baz`"
        )

The above function’s returned string is appended to the prompt template:

List five {subject}.
Your response should be a list of comma separated values, eg: `foo, bar, baz`

Based on this prompt template, calling the LLM returns a comma-separated string, such as 'red', 'orange', 'yellow', 'green', 'blue'. In the LLM chain, chain.invoke passes the LLM's text string to output_parser.invoke, which then calls output_parser.parse.

python
class CommaSeparatedListOutputParser(ListOutputParser):
    ...
    def parse(self, text: str) -> List[str]:
        """Parse the output of an LLM call."""
        return text.strip().split(", ")

The parse function uses Python's built-in split function to convert the string into an array.

Common Output Parsers in LangChain ​

Apart from CommaSeparatedListOutputParser, LangChain offers various other output parsers to meet different use cases. Below are some commonly used parsers in production environments.

DatetimeOutputParser ​

This OutputParser parses the LLM output into a Python datetime object.

Example Code:

python
from langchain_openai import OpenAI
from langchain.output_parsers import DatetimeOutputParser
from langchain.prompts import PromptTemplate

output_parser = DatetimeOutputParser()

prompt_template = PromptTemplate(
    template="{question}\n{format_instructions}",
    input_variables=["question"],
    partial_variables={"format_instructions": output_parser.get_format_instructions()},
)

model = OpenAI(temperature=0.0)
chain = prompt_template | model
output = chain.invoke({"question": "When did Hong Kong return to China?"})
# >> 1997-07-01T00:00:00.000000Z

output_parser.parse(output)
# >> datetime{1997-07-01 00:00:00}

Key Implementation of the Parser:

python
class DatetimeOutputParser(BaseOutputParser[datetime]):
    format: str = "%Y-%m-%dT%H:%M:%S.%fZ"
    def get_format_instructions(self) -> str:
        examples = comma_list(_generate_random_datetime_strings(self.format))
        return (
            f"Write a datetime string that matches the "
            f"following pattern: '{self.format}'.\n\n"
            f"Examples: {examples}\n\n"
            f"Return ONLY this string, no other words!"
        )
     
    def parse(self, response: str) -> datetime:
         ...
         return datetime.strptime(response.strip(), self.format)

The DatetimeOutputParser controls the time format using the format variable.

The get_format_instructions method generates some random date-time strings as examples, ensuring the model's output matches the expected time format. The returned string may look like:

python
from langchain.output_parsers import DatetimeOutputParser
output_parser = DatetimeOutputParser()
output_parser.get_format_instructions()
"""
Write a datetime string that matches the 
    following pattern: "%Y-%m-%dT%H:%M:%S.%fZ". 
    Examples: 1946-10-14T23:30:26.845253Z, 1392-07-26T13:26:26.557091Z, 1780-08-08T01:24:15.382408Z
"""

The parse method converts the text response from the model into a datetime object.

PydanticOutputParser ​

Pydantic is a powerful library in Python that helps developers perform field parsing and type validation. In Python, data type constraints are not enforced strictly. For instance, in a class definition like this:

python
class Member:
    def __init__(self, name: str, age: int):
        self.name = name
        self.age = age

Even if the age field is defined as an int, assigning a string to it won't raise an error:

python
member = Member(name="Jay", age="one")
print(member.age)
# > one

This flexibility increases the risk of instability and potential crashes in the program. By adding Pydantic's BaseModel to the Member class, we can enable strict type checking:

python
from pydantic import BaseModel

class Member(BaseModel):
    name: str
    age: int

member = Member(name="Jay", age="one")
"""
...
pydantic_core._pydantic_core.ValidationError: 1 validation error for Member
age
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='one', input_type=str]
"""

Now, any inconsistency between the assigned data type and the defined type raises an exception.

Introduction to PydanticOutputParser ​

PydanticOutputParser is a powerful output parser provided by LangChain. Based on the Pydantic library, it can convert the model output into custom class objects and perform custom validations to ensure the fields meet expected criteria.

Example Code ​

python
from langchain.prompts import PromptTemplate
from langchain_openai import OpenAI
from langchain.output_parsers import PydanticOutputParser
from langchain_core.pydantic_v1 import BaseModel, Field, validator

class Joke(BaseModel):
    setup: str = Field(description="question to set up a joke")
    punchline: str = Field(description="answer to resolve the joke")

    # Check that the setup field ends with a "?" (applies during parse)
    @validator("setup")
    def question_ends_with_question_mark(cls, field):
        if field[-1] != "?":
            raise ValueError("Badly formed question!")
        return field

output_parser = PydanticOutputParser(pydantic_object=Joke)

prompt_template = PromptTemplate(
    template="Tell me a joke.\n{format_instructions}",
    input_variables=[],
    partial_variables={"format_instructions": output_parser.get_format_instructions()},
)

llm = OpenAI()
chain = prompt_template | OpenAI()

output = chain.invoke({})
print(output)
# >> {"setup": "Why don't scientists trust atoms?", "punchline": "Because they make up everything."}

print(output_parser.parse(output))
# >> Joke{setup='Why don't scientists trust atoms?' punchline='Because they make up everything.'}

Key Implementation Details ​

get_format_instructions ​

When using PydanticOutputParser, a custom class inherited from BaseModel needs to be provided, specifying the expected field names and their descriptions.

In the get_format_instructions method, the schema for the custom class is obtained using BaseModel's schema method. For the Joke class in the example, the resulting schema looks like:

json
{
    "title": "Joke",
    "type": "object",
    "properties": {
        "setup": {
            "title": "Setup",
            "description": "question to set up a joke",
            "type": "string"
        },
        "punchline": {
            "title": "Punchline",
            "description": "answer to resolve the joke",
            "type": "string"
        }
    },
    "required": ["setup", "punchline"]
}

The schema is then trimmed and passed into PYDANTIC_FORMAT_INSTRUCTIONS, which has the following content:

PYDANTIC_FORMAT_INSTRUCTIONS = """
The output should be formatted as a JSON instance conforming to the following schema.
For example, for the schema {{"properties": {{"foo": {{"title": "Foo", "description": "a list of strings", "type": "array", "items": {{"type": "string"}}}}}}, "required": ["foo"]}}, the object {{"foo": ["bar", "baz"]}} is a well-formed instance of the schema. The object {{"properties": {{"foo": ["bar", "baz"]}}}} is not well-formed.
Here is the output schema:
```
{schema}
``` """

This output template controls the model's output, ensuring the returned JSON string conforms to the specified field information.

The fields in the custom class are defined using Field objects, where each field can have multiple properties affecting its final output. For example:

python
def Field(
    default: Any = Undefined,
    *,
    default_factory: Optional[NoArgAnyCallable] = None,
    alias: Optional[str] = None,
    title: Optional[str] = None,
    description: Optional[str] = None,
    exclude: Optional[Union['AbstractSetIntStr', 'MappingIntStrAny', Any]] = None,
    include: Optional[Union['AbstractSetIntStr', 'MappingIntStrAny', Any]] = None,
    const: Optional[bool] = None,
    gt: Optional[float] = None,
    ge: Optional[float] = None,
    lt: Optional[float] = None,
    le: Optional[float] = None,
    ...
)

The description attribute is crucial, as the LLM uses it to generate the corresponding information.

The PydanticOutputParser.parse method ultimately calls PydanticOutputParser.parse_result.

In PydanticOutputParser.parse_result, the model's returned JSON string is converted into the expected class object (such as a Joke object) using BaseModel.parse_obj.

StructuredOutputParser ​

While PydanticOutputParser is powerful, it can be complex. Sometimes, a simpler approach is desired for structuring output from a Language Model (LLM). In such cases, StructuredOutputParser is an ideal choice. This output parser is designed for straightforward structured output.

Example Code :

python
from langchain.output_parsers import ResponseSchema
from langchain.output_parsers import StructuredOutputParser
from langchain_openai import OpenAI
from langchain.prompts import PromptTemplate

response_schemas = [
    ResponseSchema(name="setup", description="question to set up a joke"),
    ResponseSchema(name="punchline", description="answer to resolve the joke")
]

output_parser = StructuredOutputParser.from_response_schemas(response_schemas)

prompt_template = PromptTemplate(
    template="Tell me a joke\n{format_instructions}",
    input_variables=[],
    partial_variables={"format_instructions": output_parser.get_format_instructions()},
)

chain = prompt_template | OpenAI()

output = chain.invoke({})
"""
```json
{
        "setup": "Why don't scientists trust atoms?",
        "punchline": "Because they make up everything."
}
```
"""

output_parser.parse(output)
"""
{
"setup": "Why don't scientists trust atoms?",
"punchline": "Because they make up everything."
}
"""

Key Implementation Details :

python
line_template = '\t"{name}": {type}  // {description}'
...
def _get_sub_string(schema: ResponseSchema) -> str:
    return line_template.format(
        name=schema.name, description=schema.description, type=schema.type
    )

The formatted field strings are then inserted into a template for structured format instructions:

STRUCTURED_FORMAT_INSTRUCTIONS = """
The output should be a Markdown code snippet following the pattern below, including the "```json" and "```" around it:
```json
{{
{format}
}}
```
"""

For the example provided, the final output format instructions generated would look like this:

plaintext
output_parser.get_format_instructions()
The output should be a Markdown code snippet following the pattern below, including the "```json" and "```" around it:

```json
{
        "setup": string  // question to set up a joke
        "punchline": string  // answer to resolve the joke
}
```

The parse method converts the model output from a JSON string to a Python dictionary using json.loads. It also verifies that the resulting dictionary contains the expected keys, as defined in the ResponseSchema list.

OutputFixingParser ​

The OutputFixingParser is a unique output parser that doesn't merely format the output but "fixes" formatting issues when other output parsers encounter errors. This can be helpful when an LLM generates an improperly formatted output that fails to parse using the standard parsers.

Example Scenario :

Consider the example from the StructuredOutputParser:

plaintext
llm_err_response = """
```json
{
        'setup': 'Why don't scientists trust atoms?',
        'punchline': 'Because they make up everything.'
}
```

In this case, the JSON string uses single quotes instead of double quotes, which makes it invalid. If we try parsing it using StructuredOutputParser, an exception will be thrown. This is where OutputFixingParser can be used to attempt to correct the issue.

Example Code :

python
from langchain.output_parsers import ResponseSchema, StructuredOutputParser, OutputFixingParser
from langchain_openai import OpenAI

# Constructing the output parser for expected format
response_schemas = [
    ResponseSchema(name="setup", description="question to set up a joke"),
    ResponseSchema(name="punchline", description="answer to resolve the joke")
]
structured_output_parser = StructuredOutputParser.from_response_schemas(response_schemas)

# Creating an OutputFixingParser instance for error correction
fixing_output_parser = OutputFixingParser.from_llm(parser=structured_output_parser, llm=OpenAI())

# The erroneous LLM output
llm_err_response = """
```json
{
        'setup': 'Why don't scientists trust atoms?',
        'punchline': 'Because they make up everything.'
}
```
"""

# Attempt to parse using StructuredOutputParser, which will throw an exception
try:
structured_output_parser.parse(llm_err_response)
except Exception as e:
print(f"Error: {e}")

# Using OutputFixingParser to fix the LLM's output and parse it
fixed_output = fixing_output_parser.parse(llm_err_response)
print(fixed_output)
"""
{
"setup": "Why don't scientists trust atoms?",
"punchline": "Because they make up everything."
}
"""

from_llm Method ​

The OutputFixingParser is created using the from_llm method, which requires an LLM object and a business-specific output parser (such as StructuredOutputParser). Within this method, an LLM chain is created and assigned to the retry_chain variable, while the business-specific output parser is stored in parser.

python
def from_llm(
    cls,
    llm: BaseLanguageModel,
    parser: BaseOutputParser[T],
    prompt: BasePromptTemplate = NAIVE_FIX_PROMPT,
    max_retries: int = 1,
) -> OutputFixingParser[T]:
    from langchain.chains.llm import LLMChain
    chain = LLMChain(llm=llm, prompt=prompt)
    return cls(parser=parser, retry_chain=chain, max_retries=max_retries)

get_format_instructions Method ​

This method returns the format instructions from the underlying business parser.

python
def get_format_instructions(self) -> str:
    return self.parser.get_format_instructions()

The parse method tries to parse the LLM output using the specified business parser. If it fails, the retry_chain is used to attempt a correction by generating a revised response based on the provided format instructions and the original error message. This process is repeated up to max_retries times.

python
def parse(self, completion: str) -> T:
    retries = 0
    while retries <= self.max_retries:
        try:
            return self.parser.parse(completion)
        except OutputParserException as e:
            if retries == self.max_retries:
                raise e
            else:
                retries += 1
                completion = self.retry_chain.run(
                    instructions=self.parser.get_format_instructions(),
                    completion=completion,
                    error=repr(e),
                )
    raise OutputParserException("Failed to parse")

Error Correction Process ​

The retry_chain uses a default prompt template, which attempts to guide the LLM to generate a corrected output that conforms to the specified format. The prompt template used is:

plaintext
NAIVE_FIX = """
Instructions:
{instructions}
Completion:
{completion}
The completion above did not satisfy the constraints given in the instructions. Error:
{error}
Please try again. Answer strictly following the constraints listed in the instructions:
"""

Summary ​

The OutputFixingParser adds an extra layer of robustness by wrapping around the standard output parsers. It helps handle formatting issues in LLM outputs by utilizing the LLM itself to correct the output based on the original parsing instructions. This approach improves the overall reliability and user experience, although it may come with a performance cost due to additional LLM invocations.