Description

This grant is aimed to bring AI technology to Decentraland through Open Source tools. Obviously, AI & LLMs (Large Language Models) have changed the world in the last year. We hear from all that we will not interact with the world in the old manner. That's why we need to stay updated in this field in Decentraland. This proposal outlines an advanced AI integration system with applications in virtual environments and language processing. It includes a customizable AI character, multilingual interaction capabilities, and diverse backend setups compatible with various LLMs like OpenAI and Open Source Models. Key tools such as MemGPT and AutoGen enhance AI character detailing and interaction. The system's use cases are versatile, covering virtual guides in Decentraland for navigation and information, interactive storytelling and roleplay, AI companions for social interaction, innovative educational modules, and AI-driven customer service representatives, decorations, quests, audio & speech features. Progress from the last proposal: 1. [Chat-bot for Golf Craft Game: Deployed in Genesis City](https://decentraland.org/play/?position=55%2C-42) 2. [Chat-bot for Decentral Game ICE Poker](https://twitter.com/TonyMo74/status/1745380483389956509) 3. [Multilanguage Q&A](https://twitter.com/TonyMo74/status/1746759684705632643) 4. [Image generation integration](https://twitter.com/TonyMo74/status/1740682713332637925) 5. [Retweet by LangChain](https://twitter.com/LangChainAI/status/1746569216239808885)(a widely used library for building AI/LLM applications) - They highlighted that it is a completely new interface for AI. These tweets received 55K impressions! Many likes, views, etc. Several developers (non-DCL related) asked me in DMs how to build on Decentraland + AI. This brings in completely new people to the board. Planned features (all open source): - Frontend library for AI in SDK7 (includes NPC Characters, Shapes for Image Generations, prompts for music, voice, text generation). - Backend server with abilities to use proprietary API (OpenAI, Claude2) and Local Open Sourced models. - Backend for running local models (ollama) and tools (like MemGPT, LangChain, etc). - Comprehensive documentation on how to run it. Features available: - Chat Bot NPC knowledgeable about specific projects, by uploading PDFs, links, websites. - Multilanguage Q&A Characters. - Image Generation. - Voice Generation (new level of interactivity). - Music Generation. - AI role play NPC - predefined stories and behavior by creators. - First experiments of integrating Web3 functionality to AI, including NPCs with their own Ethereum addresses for transactions.