ScreenAI：UIと視覚的状況言語理解のための視覚言語モデル

<span class="byline-author">Posted by Srinivas Sunkara and Gilles Baechler, Software Engineers, Google Research</span>

<img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoXlMR7pAKRRnyKZT8C40i6mPX0KKNGT6AFNvFOFIhZ7BD0rXaU3NS_aqISTGq9S_d0zozgcO0HR_v3R6Msm4uUDkaBFsFVx-miaDL6L0UhSz1Is8_L_iFjtvNE5OX9HX98t92b3r-rLQfJG1RrzVW354NdVUlIJVRLdQ_l4dFYa1773J-tJligdvh7QsX/s320/ScreenAI%20-%20hero.jpeg" style="display: none;" />

<p>

Screen user interfaces (UIs) and infographics, such as charts, diagrams and tables, play important roles in human communication and human-machine interaction as they facilitate rich and interactive user experiences. UIs and infographics share similar design principles and visual language (e.g., icons and layouts), that offer an opportunity to build a single model that can understand, reason, and interact with these interfaces. However, because of their complexity and varied presentation formats, infographics and UIs present a unique modeling challenge.

</p>

<a name='more'></a>

<p>

To that end, we introduce “<a href="https://arxiv.org/abs/2402.04615">ScreenAI: A Vision-Language Model for UI and Infographics Understanding</a>”. ScreenAI improves upon the <a href="https://arxiv.org/abs/2305.18565">PaLI architecture</a> with the flexible patching strategy from <a href="https://arxiv.org/abs/2210.03347">pix2struct</a>. We train ScreenAI on a unique mixture of datasets and tasks, including a novel Screen Annotation task that requires the model to identify UI element information (i.e., type, location and description) on a screen. These text annotations provide large language models (LLMs) with screen descriptions, enabling them to automatically generate question-answering (QA), UI navigation, and summarization training datasets at scale. At only 5B parameters, ScreenAI achieves state-of-the-art results on UI- and infographic-based tasks (<a href="https://x-lance.github.io/WebSRC/">WebSRC</a> and <a href="https://github.com/aburns4/MoTIF">MoTIF</a>), and best-in-class performance on <a href="https://github.com/vis-nlp/ChartQA">Chart QA</a>, <a href="https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=1">DocVQA</a>, and <a href="https://arxiv.org/abs/2104.12756">InfographicVQA</a> compared to models of similar size. We are also releasing three new datasets: <a href="https://github.com/google-research-datasets/screen_qa?tab=readme-ov-file#screen-annotation-dataset-details">Screen Annotation</a> to evaluate the layout understanding capability of the model, as well as <a href="https://github.com/google-research-datasets/screen_qa/tree/main?tab=readme-ov-file#short_answers-directory">ScreenQA Short</a> and <a href="https://github.com/google-research-datasets/screen_qa?tab=readme-ov-file#complexqa" target="_blank">Complex ScreenQA</a> for a more comprehensive evaluation of its QA capability.

</p>

<div style="line-height: 40%;">

<br />

</div>

<h2>ScreenAI</h2>

<p>

ScreenAI’s architecture is based on <a href="https://arxiv.org/abs/2209.06794">PaLI</a>, composed of a multimodal encoder block and an autoregressive decoder. The PaLI encoder uses a <a href="https://arxiv.org/abs/2010.11929">vision transformer</a> (ViT) that creates image embeddings and a multimodal encoder that takes the concatenation of the image and text embeddings as input. This flexible architecture allows ScreenAI to solve vision tasks that can be recast as text+image-to-text problems.

</p>

<p>

On top of the PaLI architecture, we employ a flexible patching strategy introduced in pix2struct. Instead of using a fixed-grid pattern, the grid dimensions are selected such that they preserve the native aspect ratio of the input image. This enables ScreenAI to work well across images of various aspect ratios.

</p>

<p>

The ScreenAI model is trained in two stages: a pre-training stage followed by a fine-tuning stage. First, self-supervised learning is applied to automatically generate data labels, which are then used to train ViT and the language model. ViT is frozen during the fine-tuning stage, where most data used is manually labeled by human raters.

</p>

<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjS1qatfLUw6BZZgkPxrv0Hx1pAPAehiF8q3kfA0BUyyPx4XXpwZRr75nYl99fTIQwLNmOHXhSBbpzHDnw6yQXZls1ZV-IE-d75jP5M02cRSZTYuU8FJBS4mubPzUPIuvcj_oqkEJcWtNWtnLmPZ3P1jJlDmc8GA1WNq00jUwl2o8gfLIIXlknrjy4z6y7Y/s1600/image6.gif" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="583" data-origin