Meta's New AI Project and Apple's Transformer Research

Introduction

Meta has recently announced a groundbreaking AI project that aims to outshine gpt4. Additionally, Apple and equal AI have collaborated to enhance the efficiency and accuracy of the Transformer architecture. In this blog, we will delve into the details of Meta's new AI project and Apple's research on Transformers.

Meta's AI Project

Meta's AI project is centered around a large multi-modal model that aims to achieve human-level performance on various professional and academic benchmarks. This includes passing a simulated bar exam or a biology Olympiad, which is quite impressive. The key factors contributing to Meta's success are their massive investment in hardware and infrastructure.

Meta has acquired a significant number of Nvidia h-100 AI training chips designed for training large-scale AI models. These chips are faster and more efficient than the ones used by open AI for gpt4. Furthermore, Meta has built new data centers worldwide to host these chips and provide sufficient computing power for their AI project. The transition from Microsoft's Azure Cloud platform to Meta's own infrastructure for AI training allows them more control, flexibility, and scalability.

Lama 2, Meta's latest AI model, was released as an open-source project. Although Lama 2 is a product of collaboration between Meta and Microsoft, Meta is developing their new model entirely using their own infrastructure. The release of Lama 2 serves multiple purposes, including showcasing their commitment to open-source and cross-collaboration, gathering feedback from developers and researchers, and creating hype for their new model.

Meta's AI Tools for Human Expressions

Meta has assembled a team to develop AI tools that can imitate human expressions. This team consists of experts in natural language processing, computer vision, speech recognition, and generative adversarial networks. They aim to design AI systems capable of understanding and producing natural language in various formats such as text, images, audio, and video.

One notable project is their Instagram chatbot with 30 different personalities. This chatbot can interact with Instagram users, provide product suggestions, gather customer details, offer discounts, and assist with customer questions. Users can select the chatbot's personality to match their brand, making it more relatable.

All of Meta's work ties into their larger plan to develop a metaverse, a digital space where people can engage with each other and online content in a lifelike manner. Meta sees AI as a crucial component for the metaverse, providing the intelligence and innovation to shape digital material. They believe the metaverse will revolutionize social media, shopping, entertainment, and learning.

Meta's Open Source Initiative

One of the most controversial aspects of Meta's new AI project is their decision to make the model open source and freely available for research and commercial use. This move deviates from the norm, as many big AI models are typically kept private or have usage restrictions.

Mark Zuckerberg, the head of Meta, aims to encourage creativity, equal access, transparency, and healthy competition in the AI world. By sharing their model, Meta hopes to benefit more people and foster collaboration among AI creators. Making the AI field more transparent and trustworthy is also a key objective.

While the noble sentiment of promoting collaboration and accessibility is emphasized, there are additional motivations behind Meta's decision. By making the AI model open source, Meta can leverage the expertise of AI professionals worldwide to enhance the model. This not only boosts the company's image but also paves the way for new tools, services, and partnerships.

The U.S. Senate AI Discussion

On September 13th, U.S. Senate Majority Leader Chuck Schumer is organizing an AI discussion involving key tech CEOs such as Elon Musk, Sam Altman, Sundar Pichai, Jensen Huang, Eric Schmidt, Satya Nadella, and Mark Zuckerberg. This event aims to delve into major AI regulatory issues, including safeguarding workers, national security, copyrights, preventing worst-case AI outcomes, and maintaining fairness and responsibility in AI.

This event could serve as an opportunity for Meta to present their latest AI and address potential critiques. It is also a chance for them to shape future U.S. AI policies and regulations.

Apple and Equal AI's Transformer Research

Transformers have become the hottest trend in natural language processing (NLP), especially in machine translation. They have achieved state-of-the-art results on various benchmarks. However, scaling up transformer models increases computational requirements, which poses challenges for real-world applications.

A team of researchers from Apple and Equal AI has focused on improving the efficiency and accuracy of the Transformer architecture. They specifically examined the feed forward network (FFN), a crucial component of Transformers responsible for transforming each input token independently. The researchers discovered that the FFN exhibits a high level of redundancy, consuming a significant number of parameters.

The researchers made adjustments to the Transformer architecture by using just one FFN for all the encoder layers instead of having it in the decoder layers. This reduced the number of parameters without compromising accuracy. They also adjusted the shared FFN's hidden dimension, making the FFN wider but not as deep. This optimization made the FFN more efficient and easier to use for various language tasks.

Apple and Equal AI's research highlights the possibilities for enhancing the Transformer architecture's design and reveals the untapped potential in language processing and AI.

Conclusion

Meta's new AI project and Apple's Transformer research signify significant advancements in the field of artificial intelligence. Meta's investment in hardware and infrastructure, along with their open-source initiative, demonstrates their commitment to innovation, collaboration, and accessibility in the AI community. Apple and Equal AI's research on optimizing Transformers showcases the ongoing quest for more efficient and practical AI models. The future of AI holds great promise, and these developments pave the way for further exploration and progress.