Grok-1.5 ビジョンプレビュー

Grok-1.5 Vision Preview | xAIApril 12, 2024Grok-1.5 Vision Preview

Connecting the digital and physical worlds with our first multimodal model.

Introducing Grok-1.5V, our first-generation multimodal model. In addition to its strong text capabilities, Grok can now process a wide variety of visual information, including documents, diagrams, charts, screenshots, and photographs. Grok-1.5V will be available soon to our early testers and existing Grok users.

Grok-1.5V is competitive with existing frontier multimodal models in a number of domains, ranging from multi-disciplinary reasoning to understanding documents, science diagrams, charts, screenshots, and photographs. We are particularly excited about Grok’s capabilities in understanding our physical world. Grok outperforms its peers in our new RealWorldQA benchmark that measures real-world spatial understanding. For all datasets below, we evaluate Grok in a zero-shot setting without chain-of-thought prompting.

Diagrams88.3%78.2%88.7%88.1%80.3%TextVQAText reading78.1%78.0%--73.5%ChartQACharts76.1%78.5%81.1%80.8%81.3%DocVQADocuments85.6%88.4%89.5%89.3%86.5%RealWorldQAReal-world understanding68.7%61.4%51.9%49.8%67.5% ExampleWriting code from a diagramUser

Certainly! The flowchart you’ve provided describes a simple guessing game where the computer generates a random number, and the user has to guess it. Here’s the Python code that represents the logic in the flowchart:

import random def guess_number(): # Generate a random number between 1 and 10 target = random.randint(1, 10) while True: guess = int(input("Guess a number between 1 and 10: ")) if guess == target: print("You won!") break else: print("Wrong guess, try again!") if __name__ == "__main__": guess_number()This code will run the guessing game as described in the flowchart. The user will be prompted to enter their guess until they guess the correct number.

In order to develop useful real-world AI assistants, it is crucial to advance a model's understanding of the physical world. Towards this goal, we are introducing a new benchmark, RealWorldQA. This benchmark is designed to evaluate basic real-world spatial understanding capabilities of multimodal models. While many of the examples in the current benchmark are relatively easy for humans, they often pose a challenge for frontier models.

Which object is larger the pizza cutter or the scissors? A. The pizza cutter is larger. B. The scissors is larger. C. They are about the same size.

Where can we go from the current lane? A. Turn left. B. Go straight. C. Turn left and go straight. D. Turn right.

Given this front camera view from our sedan, do we have enough space to drive around the gray car in front of us? A. Yes. B. No.

Given the picture, in which cardinal direction is the dinosaur facing? A. North. B. South. C. East. D. West.

The initial release of the RealWorldQA consists of over 700 images, with a question and easily verifiable answer for each image. The dataset consists of anonymized images taken from vehicles, in addition to other real-world images. We are excited to release RealWorldQA to the community, and we intend to expand it as our multimodal models improve. RealWorldQA is released under CC BY-ND 4.0. Click here (677MB) to download the dataset.

Advancing both our multimodal understanding and generation capabilities are important steps in building beneficial AGI that can understand the universe. In the coming months, we anticipate to make significant improvements in both capabilities, across various modalities such as images, audio, and video.

If you want to be a part of this journey, we are hiring.