Microsoft Launches AI Cosmos 2: Taking Multimodal AI to a New Level

Introduction

Microsoft has recently unveiled its latest AI model, Cosmos 2, which brings a new level of interaction and understanding to artificial intelligence. This groundbreaking model not only allows for easier ways to chat with AI using pictures instead of long text, but also enables AI to understand and respond to images just like humans do. In this blog, we will explore the concept of multimodal AI, the limitations of previous models, and how Cosmos 2 overcomes these limitations with its new feature called grounding.

Multimodal AI and Its Importance

Multimodal AI is a type of artificial intelligence that combines different kinds of data such as text, images, videos, and sounds. Its purpose is to build AI systems that can understand and create content from various sources, similar to how humans do. In the past, AI systems could only handle one type of data at a time, but with the development of multimodal large language models (MLMs), such as Cosmos 2, AI can now process multiple types of data simultaneously and generate mixed content.

Microsoft introduced its first multimodal language model, Cosmos 1, which excelled at tasks like writing stories from images, creating image captions, and answering questions about images. However, Cosmos 1 had limitations in understanding and connecting visual information. It struggled to grasp the meaning of images or link them to other data types. For example, it could generate text based on an image, but couldn't point to specific areas in the image using words or coordinates, nor answer questions requiring visual reasoning.

Introducing Cosmos 2 and Grounding

Cosmos 2, the latest version of Microsoft's MLM, introduces a feature called grounding that allows for more accurate and meaningful interaction with images. Grounding works by creating hyperlinks between parts of an image and its description, enabling the model to connect images with words. This feature makes it easier to match words with specific parts of a picture, enhancing the understanding between humans and AI.

Imagine a picture as a checkerboard, with each square assigned a special tag. When you click on a part of the image, Cosmos 2 highlights the corresponding word or sentence that describes it. Similarly, when given a token like "a," the model can predict an image token (IMG) and generate an image based on it. Cosmos 2 was trained on extensive multimodal data, allowing it to convert images into tokens, generate text from images, and use location tokens for grounding.

Performance and Practical Applications of Cosmos 2

Cosmos 2 excels in various tasks, such as identifying phrases and images, processing language, and outperforming other models. For example, in locating phrases in images, it achieves an accuracy of 91.3%, compared to other models' scores of 78.4% and 86.7%. However, what makes Cosmos 2 truly special goes beyond its performance.

Some practical applications of Cosmos 2 include:

• Grounded picture captioning: Cosmos 2 can generate detailed captions for images, marking specific regions with location tokens. This benefits people with visual impairments, helps students learn new concepts, and enables content creators to craft more immersive stories.
• Grounded visual question answering: It can answer questions about specific regions within images, denoted by location tokens. This is useful for users seeking detailed information, researchers extracting insights, or customers making image-based decisions.
• Grounded visual reasoning: It can perform logical or mathematical operations based on specific regions in an image. This has potential in solving image-based puzzles, teaching mathematical skills, or creating challenges for game developers.

Cosmos 2 has countless uses and benefits tailored to individual needs, and its potential is limitless. To experience its capabilities firsthand, Microsoft has released an online demo of Cosmos 2 on GitHub. The demo allows users to interact with the model, test its features, and compare its performance with other MLMs or unimodal models. It is easy to use and provides a fun way to explore different scenarios and tasks.

Try out the Cosmos 2 demo here and share your feedback and suggestions in the comments section.

Conclusion

Microsoft's AI Cosmos 2 takes multimodal AI to new heights by enabling AI to interact with images more accurately and meaningfully. With its grounding feature, Cosmos 2 bridges the gap between images and words, allowing for human-like interaction. Its performance surpasses other models, and its practical applications are numerous, ranging from assisting visually impaired individuals to aiding researchers and content creators. The online demo provides an opportunity to explore Cosmos 2 and witness its transformative potential. Don't miss out on this exciting advancement in AI technology!