Team
Jim, Serhad and Maxi
Concept
This is an RPG battling robots game for the Quest 3 visor from Meta. It involves heavy use of the Llama AI model. We use Quest hand and voice commands. NPC characters can hold conversations on most any topic with AI generated spoken dialog.
Story
Our story takes place on Mars in the future. Our player is the pilot of a giant fighting robot. Robots with human pilots fight each other for the entertainment of a television audience.
The pilots are prisoners. If they win a match they get a cash prize which they can use to buy better weapons. If they win a certain number of matches they will go free. There may be other ways to get free. Perhaps there is a way to escape or buy your way out.
Imagine the movie "Spartacus", but in the future, with robots and crazy high tech weapons. This will give you an idea of the world we would like to build.
Note: Spartacus (1960) - video Dailymotion (see it for free, the movie is one of the best Roman epics ever made. There is three minutes of music at the front and then the story about the famous gladiators revolt in ancient Rome.)
There are also NPC characters who sell weapons, arrange matches, coaches, repairmen, tv announcers, and even fans.
Pilots of the other mechs (NPCs) are historical characters such as Shakespeare, Napoleon, and Blackbeard the Pirate. They have somehow been reanimated and put in modern bodies. Their challenge now is to survive and get free.
All players (NPCs) can talk and respond to voice commands and speak in the voice/style of their character. For example, our pirate NPC likes to say "Arrh Matey" and "Shiver me timbers!" and other silly pirate talk. Likewise Shakespeare spouts Elizabethan prose while piloting his robot.
Game Platform - Quest 3
The game is programmed in Unity for Quest 3 and Windows.
Animations are done in Maya.
Character dialog is created using Llama 3.1
The game uses sound (speaker and microphone required)
Hand motions are detected and used with or without controllers.
Game Assets
We have created original animatable giant robots in Autodesk's Maya.
We will add more types of weapons, props and scenery later.
| Robot Type 1 | Robot Type 2 |
|---|---|
|
|
Pilot characters are from Mixamo
Voice and AI
Our game design makes extensive use of the AI model Llama 3.1 which is used to resolve character interactions and generate interesting character dialogue.
It will work something like this:
- Player says something
- Game converts this to text.
- AI determines if it is a communication, or a command. It a spoken item is a command, the AI will classify it as a move, a combat action, a system command such as "exit game", and also will determine if the activity is allowed.
- If the spoken item is a communication, the AI will generate a response in text format which the game will then voice.
- All commands will have controller and keyboard versions, so even if you don't have a Quest 3, the game will work.
Here is a link to a demonstration Maxi made of how the voice recognition works in Unity:
https://youtube.com/shorts/ismEtPwuqOE?feature=share
Sound Design
We will need good sounds and good music for our game.
All of the music will be AI generated.
Here's an old project I (Jim) made in another class that will give an idea of the proposed sound design style:
https://youtu.be/KSXJAu7AM3Y
Theme - Chain Reactions
There will be some barrels on the fight arena. Different kind of barrels have different effects on the robots. We will hit those barrels to create different kind of effects and make some chain reactions.
Modification: We are considering adding mines as a weapon in the arena. A player could set one mine every thirty seconds. A player laying a mine would not be able to move at the same time.
The mines would be as follows:
- Standard exploding mine, enough to do 1/3 damage to a robot
- Immobilizing mine, will stop robot for three minutes or until robot repaired
- Radio blocker mine: if you are near one of these your opponent can't see you.
- Mine sweeper non mobile device that will one by one destroy nearby mines
Milestone 0
Milestone 1: Formal Game Proposal
Document:
Slides:
Milestone 2: Prototype
Document:
Slides:
Milestone 3: Interim Report
Document:
Slides:
28 Kommentare
Lukas Moersler sagt:
24. Oktober 2024Favorite aspect: VR mech fight concept
Least favorite aspect: Insane scope of the project
Suggestion: The barrels sound very secondary to the actual gameplay right now, I'd suggest integrating them a bit more. Maybe let players launch their own barrels or move existing ones around before setting them off to create the chains in the first place. With VR you have the chance to let users move the barrels far more intuitively and accurately than any 2D interface
Daniel Abert sagt:
24. Oktober 2024favourite aspect:
It'll be interesting to have the opportunity to fight against Shakespeare and other personalities und to talk to them.
least favourite aspect:
The scope seems very big for the given time period. The ideas on their own seem pretty cool, but I don't yet see how they synergize together.
suggestions:
It'd be cool, if the conversations (e.g. insults) would influence the fights.
James Pickrell sagt:
30. Oktober 2024Yeah I have been reading about the insult swordfighting.
Such a cool idea.
I need to think about how to integrate this because it could really add an unexpected dimension to the combat.
Felix Wenzel sagt:
24. Oktober 2024Favorite Aspect:
Insulting Napolean ;D
No jokes aside i like the possibility of interacting with the ai chars, and having it influence the game.
Least favourite Aspect:
You have great ideas, but the chain reaction aspect seems a little forced. maybe you can find a better fitting one.
Suggestion:
Use the barrel aspect more, and let them be essential gameplay, otherwise it just seems forced.
James Pickrell sagt:
25. Oktober 2024The ai characters thing works very well. This part of the game is guaranteed to be good. You will be totally impressed. I don't know what we are going to do about lip synch and accents when the audio comes out, but the dialogue is very good.
James Pickrell sagt:
25. Oktober 2024I will bring a demo to class.
Christian Barbu sagt:
24. Oktober 2024Favorite aspect: LLM-based conversations, VR.
Least favorite aspect: Story seems a bit arbitrary.
Suggestion: Focus on one aspect and do it well (as it was mentioned by the instructors). There is no need for a fight-scenario, if you incorporate LLM and vice versa. It would be nice to see a small interactable world, where there is a small story, which leads the player through it and the companions are LLM based. The LLM chars could be given tasks in the limited 3D environment (however this may not be the focus). The chain reaction aspect could be given by making use of decision-based branching.
James Pickrell sagt:
27. Oktober 2024We are not really using deterministic branching. The game has a large randomness effect and depends largely on player actions and what the player does and says. You can offer Napoleon $50 to throw a match. Will he take it? And so on.
Natalia Piotrowski sagt:
24. Oktober 2024Favorite aspect: LLM-based conversations
Least favorite aspect: The LLM doesnt seem to fit that naturally into your game as of right now.
Suggestion: You could integrated the LLM into the fighting mechanics and also maybe implement your chain reactions in that way. Like in the beginning of each round you are handed a sort of overview over who you will be fighting alongside some information about them (i would refrain from using real characters for this part). Your task would then be to "chain" together the right insults/comments to make the enemy quit the fight.
James Pickrell sagt:
27. Oktober 2024We intend to have two parts to the game. One part is during matches. The other part is between the matches. At that time the player can conspire with other players, attempt to bribe officials, and buy better weapons. Imagine Napoleon and Asterix and Trump and John Carter from Barsoom all in jail together trying to organize a revolt. Is it possible? Because of our small amount of manpower I don't know how much scenery we can build but we will give it a try.
Melih Yilmaz sagt:
24. Oktober 2024Favorite aspect: I like the idea of battling enemies with robots (while in VR).
Least favorite aspect: I am not sure about the use of AI for conversations. I will have to see how it incorporates into the gameplay or if it is just used for the sake of using AI.
Suggestion: The project seems very ambitious, so I would focus on getting the fighting and controls right, before doing other stuff like alternative winning methods, buying weapons etc.
Miguel Trasobares Baselga sagt:
24. Oktober 2024Favorite aspect: Using an LLM to give characters unique interactions with the player seems very interesting, both because nothing would be 100% scripted and it gives you a wide range of possible options for interaction.
Least favorite aspect: The actual fight mechanics seem a bit forgotten, if the player is gonna spend most of the time fighting with robots I am not quite sure if the conversation mechanic can compensate the core gameplay.
Suggestion: Integrating the conversation mechanic into the game, not only as something cosmetic but with influence in the game would be awesome. Maybe you can add some interaction by talking with your opponent trying to fool him, with the crowd giving you cheers or maybe with the organizers for dropping bonus consumables in the arena (here I am thinking like in "The Hunger Games" were participants get help from outside depending on audience and so on). Maybe you can also consider talking with some AI in the robot to activate perks/weapons/etc.
James Pickrell sagt:
27. Oktober 2024Agree on the importance of the voice mechanics.
Haorui Tan sagt:
24. Oktober 2024Favorite aspect: VR interaction is very interesting
Least favorite aspect: Dialogue might become repetitive over time if the AI responses lack enough variety.
Suggestion: Introducing a system where NPCs' personalities evolve over time based on their interactions with the player would add depth and variety to the dialogue, making each encounter feel more unique.
James Pickrell sagt:
27. Oktober 2024The character dialog is different every time.
We are working to make sure that the character dialog is consistent over time. We dont want them to contradict themselves.
Jialin Yang sagt:
24. Oktober 2024Favorite aspect: Using LLMs to mimic famous characters is a good idea.
Least favorite aspect: Speech-to-text input may not be that reliable. Imagine playing the game with a TV on...
Suggestion: Maybe give more focus on chain reaction? Now it seems that taking advantage of chain reaction isn't in the core of the game.
James Pickrell sagt:
27. Oktober 2024The characters are hilarious.
I hope the rest of the game will be as fun.
Yifei Liu sagt:
24. Oktober 2024Favorite aspect: No one can resist the temptation to pilot a mech.
Expectation: If conditions allow, could a feature be added that enables the mech to briefly fly or glide through a jump? It might not be very useful, but I simply want to experience mech flying in VR.
James Pickrell sagt:
25. Oktober 2024I like it. I will try to make this happen.
James Pickrell sagt:
25. Oktober 2024How about an afterburners button? You press the button and whooosh kaboom you jump over your opponent and put him at disadvantage...
Mahdis Sabzevarzadeh sagt:
25. Oktober 2024Favorite aspect I really like the aspect of creating NPC characters as a mix of historical and fictional figures, I'm very interested to interact with different NPCs and see their dialogs
Least favorite aspect The fighting perspective with use of controller free hand-tracking system doesn't seem very interesting to me for a game whose objective is to fight in an arena
Suggestion I would focus fighting mechanics to be done with a controller/mouse and keyboard. Maybe some other aspects of the game like interaction with NPC or menu can be done by Meta 3
James Pickrell sagt:
27. Oktober 2024The historical character simulations are hilarious.
They are not always accurate on facts and some of what they say is pure hallucination, but it is funny.
Yaxuan Dai sagt:
25. Oktober 2024Favorite aspect: The characters introduced with their own background. This might double my playing time to explore setting of different characters.
Least favorite aspect: I thought this is a VR/AR game according to your selection of device. But I have no idea how could this be corporate with the setting that your robot is giant. Are there any connection?
Suggestion: If the interaction with NPCs is via audio, I would suggest testing with noise disturbance from the first version for the sake of the final demo.
James Pickrell sagt:
27. Oktober 2024So far, it works.
Non-English speaking accent is also a concern.
Our tests have been simple but so far it works.
Cynthia Drews sagt:
25. Oktober 2024Most favorite aspect:
Differently styled NPCs! I really wish they all have different fighting styles too! Or special abilities that they can use like in Overwatch or LoL.
Least favorite aspect:
I can't imagine the barrel aspect having much of an impact yet. I think more complex maps that aren't just planar could help with that. Quake had some cool maps with different floors.
Suggestions:
Will the player be able to channel communication to other players or NPCs, or will it only be broadcast in a specific area? And is it more push-to-talk or activation by voice? I know you check if "Exit the game" is possible in the current context if it's accidentally said in the middle of the game, but it could also be a disadvantage if the AI is always listening. The same goes for gestures. If a friend comes over and starts talking to you while you're showing them the game, and that would interfere with your game, that would be a problem. Or if someone wants to stream their game and entertain an audience.
James Pickrell sagt:
27. Oktober 2024These are good questions on the communication.
I was kind of thinking, in person maybe 10 meters range.
While in robot, the radio has infinite range.
As to when the microphone is on, that is also a good question. We will have to try it and see what works.
I think it is important that the player has control of when he broadcasts or speaks to other players.
So the player could trigger the microphone with one clap for on and one clap for off. Or press a certain button on the controller. Or say "Talk" and "Done talking."
Andrey Roytman sagt:
26. Oktober 2024Most favorite aspect: text-to-speech mechanic.
Least favorite aspect: controller-free robot controls in combat.
James Pickrell sagt:
27. Oktober 2024Using the hands-only control feature is a matter of choice. We will also support Quest controller and mouse. The user can choose.