> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vlm.run/llms.txt
> Use this file to discover all available pages before exploring further.

# UI Parsing

> Analyze and understand user interface elements in screenshots and application images

Analyze and understand user interface elements in screenshots and application images. Perfect for automated testing, design system validation, accessibility auditing, and mobile app analysis.

<Frame caption="UI parsing example showing element detection and classification with interactive elements">
  <img src="https://mintcdn.com/autonomiai/vXlBwf-F5SFbY_zw/agents/assets/ui-parsing.jpg?fit=max&auto=format&n=vXlBwf-F5SFbY_zw&q=85&s=ba176b12357cb8279b4902e7c4e41f03" alt="UI parsing example showing UI element detection and classification with interactive elements" width="1708" height="960" data-path="agents/assets/ui-parsing.jpg" />
</Frame>

<Frame caption="Examples of UI parsing for different interface types.">
  <div style={{ display: 'flex', gap: '20px', justifyContent: 'center' }}>
    <div style={{ flex: 1 }}>
      <div style={{ textAlign: 'center', marginBottom: '10px' }}>UI VQA & Grounding</div>

      <img src="https://mintcdn.com/autonomiai/vXlBwf-F5SFbY_zw/agents/assets/ui-shadcn-vqa.jpg?fit=max&auto=format&n=vXlBwf-F5SFbY_zw&q=85&s=5bf5dbce3518663eadccceb6c93948dd" alt="UI VQA & Grounding" style={{ width: '100%', border: '1px solid #ddd', borderRadius: '4px' }} width="1440" height="840" data-path="agents/assets/ui-shadcn-vqa.jpg" />
    </div>

    <div style={{ flex: 1 }}>
      <div style={{ textAlign: 'center', marginBottom: '10px' }}>Web Interface</div>

      <img src="https://mintcdn.com/autonomiai/vXlBwf-F5SFbY_zw/agents/assets/ui-parsing-vqa.jpg?fit=max&auto=format&n=vXlBwf-F5SFbY_zw&q=85&s=bb9d3fd32117159283bd80a1f604fefd" alt="Web interface UI parsing" style={{ width: '100%', border: '1px solid #ddd', borderRadius: '4px' }} width="1440" height="840" data-path="agents/assets/ui-parsing-vqa.jpg" />
    </div>
  </div>
</Frame>

## Usage Example

<Tip>
  For best results, we recommend using the [Structured Outputs API](/agents/structured-responses) to get responses in a structured and validated data format.
</Tip>

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  from vlmrun.client import VLMRun

  # Initialize the VLMRun client
  client = VLMRun(api_key="<VLMRUN_API_KEY>")

  # Parse UI elements in the image
  response = client.agent.completions.create(
      model="vlmrun-orion-1:auto",
      messages=[
          {
            "role": "user",
            "content": [
              {"type": "text", "text": "Analyze all UI elements in this mobile app screenshot"},
              {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", "detail": "auto"}}
            ]
          }
      ],
  )

  print(response.choices[0].message.content)
  ```

  ```python Python - Structured Outputs theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  from vlmrun.client import VLMRun
  from pydantic import BaseModel, Field

  # Define the response schema
  class UIElement(BaseModel):
    type: str = Field(..., description="Type of UI element")
    text: str | None = Field(None, description="Text content of the element")
    interactive: bool = Field(..., description="Whether the element is interactive")
    xywh: tuple[float, float, float, float] = Field(..., description="Bounding box coordinates")

  class UIResponse(BaseModel):
    elements: list[UIElement] = Field(..., description="List of detected UI elements")

  # Initialize the VLMRun client
  client = VLMRun(api_key="<VLMRUN_API_KEY>")

  # Parse UI elements with structured output
  response = client.agent.completions.create(
      model="vlmrun-orion-1:auto",
      messages=[
          {
            "role": "user",
            "content": [
              {"type": "text", "text": "Analyze all UI elements in this mobile app screenshot"},
              {"type": "image_url", "image_url": {"url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", "detail": "auto"}}
            ]
          }
      ],
      response_format={"type": "json_schema", "schema": UIResponse.model_json_schema()},
  )

  # Validate the response
  result = UIResponse.model_validate_json(response.choices[0].message.content)
  # >>> UIResponse(elements=[UIElement(type="button", text="Sign In", ...), ...])
  ```

  ```typescript Node.js theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { VlmRun } from "vlmrun";

  const client = new VlmRun({
    apiKey: "<VLMRUN_API_KEY>",
    baseURL: "https://api.vlm.run/v1"
  });

  const response = await client.agent.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "Analyze all UI elements in this mobile app screenshot" },
          { type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", detail: "auto" } }
        ]
      }
    ]
  });

  console.log(response.choices[0].message.content);
  ```

  ```typescript Node.js - Structured Outputs [expandable] theme={"theme":{"light":"github-light","dark":"dark-plus"}}
  import { VlmRun } from "vlmrun";
  import { z } from "zod";
  import { zodToJsonSchema } from "zod-to-json-schema";

  // Define the response schema with Zod
  const UIResponseSchema = z.object({
    elements: z.array(z.object({
      type: z.string().describe("Type of UI element"),
      text: z.string().nullable().describe("Text content of the element"),
      interactive: z.boolean().describe("Whether the element is interactive"),
      xywh: z.array(z.number()).describe("Bounding box coordinates")
    })).describe("List of detected UI elements")
  });

  // Initialize the VLMRun client
  const client = new VlmRun({
    apiKey: "<VLMRUN_API_KEY>",
    baseURL: "https://api.vlm.run/v1"
  });

  // Parse UI elements with structured output
  const response = await client.agent.completions.create({
    model: "vlmrun-orion-1:auto",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: "Analyze all UI elements in this mobile app screenshot" },
          { type: "image_url", image_url: { url: "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/web.ui-automation/win11.jpeg", detail: "auto" } }
        ]
      }
    ],
    response_format: {
      type: "json_schema",
      schema: zodToJsonSchema(UIResponseSchema)
    }
  });

  const result = UIResponseSchema.parse(JSON.parse(response.choices[0].message.content));
  ```
</CodeGroup>

## FAQ

<AccordionGroup>
  <Accordion title="What is UI Parsing?" icon="window-maximize">
    UI Parsing is the process of analyzing UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing.
  </Accordion>

  <Accordion title="What is UI VQA & Grounding?" icon="window-maximize">
    UI VQA & Grounding is the process of asking specific questions about the UI elements in screenshots and application images to identify UI elements, buttons, and interactive components for automated testing. This is different from UI parsing, where all UI elements are returned. In most cases, you should use UI VQA & Grounding to get more accurate results.
  </Accordion>
</AccordionGroup>