Team
Maximilian Amougou, Serhad Çalışkan and Jim Pickrell
Concept
This is an RPG battling robots game for the Meta Quest 3.
The game is played without controllers by using hand gestures and voice.
It involves heavy use of LLM to enhance the interactions with the Npcs.
NPC characters can hold conversations on most any topic with AI generated spoken dialog.
Furthermore the LLM is also used to steer the fight opponent and evaluate the Karma System.
The conversations are also powered with text-to-speech and speech-to-text, to make conversations feel more immersive & natural.
Story
Set on the harsh and unforgiving prison planet of Mars, players take on the role of a skilled pilot commanding a powerful combat robot. In a brutal and high-stakes competition, human pilots battle in colossal machines for a shot at freedom. Each victory earns a cash prize that can be used to upgrade weapons, enhance skills, and prepare for even tougher challenges. Adding to the intrigue, the player's opponents and NPCs are legendary figures from history, philosophy, and fiction—such as Billy the Kid, Socrates, and Severus Snape. These characters each bring their own personalities, strategies, and wisdom to the battlefield and interactions, making every encounter unique. The ultimate goal: win enough battles to secure freedom and escape the prison’s deadly chain.
All players (NPCs) can talk and respond to voice commands and speak in the voice/style of their character. For example, our pirate NPC likes to say "Arrh Matey" and "Shiver me timbers!" and other silly pirate talk. Likewise Shakespeare spouts Elizabethan prose while piloting his robot.
Game Platform - Quest 3
Our game is developed using Unity, optimized for the Meta Quest 3.
3D assets, including the robots and arenas, are modeled and animated in Maya. Interactive elements such as character dialogue, shop interactions, Karma evaluation, and opponent AI are powered by GPT-4o, bringing dynamic and engaging experiences to the player. Text-to-Speech and Speech-to-Text capabilities are integrated through Wit.ai, enhancing communication and immersion. The game leverages the Quest's built-in speaker and microphone for sound, while hand and body motions are detected and used for control without the need for controllers, offering a more natural and immersive experience.
Game Assets
We have created original animatable giant robots in Autodesk's Maya.
| Robot Prototype 1 | Robot Prototype 2 |
|---|---|
|
|
Pilot characters are taken from Mixamo
Voice and AI
Our game design makes extensive use of an LLM which is used to resolve character interactions, generate interesting character dialogue, choose the npc attacks and also to update our the Karma system (our interpretation of the theme "chain" reaction).
Here is a link to a demonstration Maxi made of how the voice recognition works in Unity:
https://youtube.com/shorts/ismEtPwuqOE?feature=share
Theme - Chain Reaction
Our interpretation of the "Chain Reaction" theme is deeply integrated into both gameplay and narrative. The player's interactions with NPCs directly influence their Karma score. This score, in turn, affects how NPCs perceive and assist the player, including shopkeepers’ willingness to negotiate prices and offer help. A high Karma score not only improves trade and combat abilities but also unlocks valuable information. NPCs can provide crucial details about upcoming fights—revealing enemy weaknesses, tactics, or hidden advantages. However, if the player's Karma is low, NPCs may withhold information or even mislead them, making battles significantly harder. Additionally, the theme extends to the visual design of the game: our projectiles take the form of chains that explode in a cascading "chain reaction," further reinforcing the concept. Beneath the surface, every action sets off a ripple effect, making each decision and outcome feel connected.
Game Mechanics
- Turn-Based Combat – Battles are round-based, with players choosing from five actions: Load, Shoot, Dodge (success chance influenced by Karma), Disarm (success chance influenced by Karma), and Shield (limited uses). Each action is executed by pressing a button in the robot’s control room.
- Upgrades & Progression – Players can enhance their robot in the shop by increasing damage & improving success probabilities for various actions.
- Karma System – Karma is influenced by the player's politeness. Even in prison, manners matter—rudeness leads to chaos, while respect earns advantages.
- NPC Interaction – Speak with nearby NPCs by making a fist, activating the microphone for a more immersive and natural dialogue experience.
- Opponent AI Difficulty – Three AI difficulty levels: Basic – Ignores game stats. Mid – Analyzes game stats. Hard – Analyzes game stats and fight history for smarter decisions
Milestone 0
Milestone 1: Formal Game Proposal
Document:
Slides:
Milestone 2: Prototype
Document:
Slides:
Milestone 3: Interim Report
Document:
Slides:
Milestone 4: Alpha Release
Document:
Slides:
Milestone 5: Game Testing
Document:
Slides:
Milestone 6: Final Release:
Document:
Slides:
Trailer:
.apk:
Note: To play the game you have to create an API key from OpenAI and get a new build.
Github Repository:
https://github.com/serhadcaliskan/GiantRobots
Demo day poster:
1-minute pitch:
28 Kommentare
Lukas Moersler sagt:
24. Oktober 2024Favorite aspect: VR mech fight concept
Least favorite aspect: Insane scope of the project
Suggestion: The barrels sound very secondary to the actual gameplay right now, I'd suggest integrating them a bit more. Maybe let players launch their own barrels or move existing ones around before setting them off to create the chains in the first place. With VR you have the chance to let users move the barrels far more intuitively and accurately than any 2D interface
Daniel Abert sagt:
24. Oktober 2024favourite aspect:
It'll be interesting to have the opportunity to fight against Shakespeare and other personalities und to talk to them.
least favourite aspect:
The scope seems very big for the given time period. The ideas on their own seem pretty cool, but I don't yet see how they synergize together.
suggestions:
It'd be cool, if the conversations (e.g. insults) would influence the fights.
James Pickrell sagt:
30. Oktober 2024Yeah I have been reading about the insult swordfighting.
Such a cool idea.
I need to think about how to integrate this because it could really add an unexpected dimension to the combat.
Felix Wenzel sagt:
24. Oktober 2024Favorite Aspect:
Insulting Napolean ;D
No jokes aside i like the possibility of interacting with the ai chars, and having it influence the game.
Least favourite Aspect:
You have great ideas, but the chain reaction aspect seems a little forced. maybe you can find a better fitting one.
Suggestion:
Use the barrel aspect more, and let them be essential gameplay, otherwise it just seems forced.
James Pickrell sagt:
25. Oktober 2024The ai characters thing works very well. This part of the game is guaranteed to be good. You will be totally impressed. I don't know what we are going to do about lip synch and accents when the audio comes out, but the dialogue is very good.
James Pickrell sagt:
25. Oktober 2024I will bring a demo to class.
Christian Barbu sagt:
24. Oktober 2024Favorite aspect: LLM-based conversations, VR.
Least favorite aspect: Story seems a bit arbitrary.
Suggestion: Focus on one aspect and do it well (as it was mentioned by the instructors). There is no need for a fight-scenario, if you incorporate LLM and vice versa. It would be nice to see a small interactable world, where there is a small story, which leads the player through it and the companions are LLM based. The LLM chars could be given tasks in the limited 3D environment (however this may not be the focus). The chain reaction aspect could be given by making use of decision-based branching.
James Pickrell sagt:
27. Oktober 2024We are not really using deterministic branching. The game has a large randomness effect and depends largely on player actions and what the player does and says. You can offer Napoleon $50 to throw a match. Will he take it? And so on.
Natalia Piotrowski sagt:
24. Oktober 2024Favorite aspect: LLM-based conversations
Least favorite aspect: The LLM doesnt seem to fit that naturally into your game as of right now.
Suggestion: You could integrated the LLM into the fighting mechanics and also maybe implement your chain reactions in that way. Like in the beginning of each round you are handed a sort of overview over who you will be fighting alongside some information about them (i would refrain from using real characters for this part). Your task would then be to "chain" together the right insults/comments to make the enemy quit the fight.
James Pickrell sagt:
27. Oktober 2024We intend to have two parts to the game. One part is during matches. The other part is between the matches. At that time the player can conspire with other players, attempt to bribe officials, and buy better weapons. Imagine Napoleon and Asterix and Trump and John Carter from Barsoom all in jail together trying to organize a revolt. Is it possible? Because of our small amount of manpower I don't know how much scenery we can build but we will give it a try.
Melih Yilmaz sagt:
24. Oktober 2024Favorite aspect: I like the idea of battling enemies with robots (while in VR).
Least favorite aspect: I am not sure about the use of AI for conversations. I will have to see how it incorporates into the gameplay or if it is just used for the sake of using AI.
Suggestion: The project seems very ambitious, so I would focus on getting the fighting and controls right, before doing other stuff like alternative winning methods, buying weapons etc.
Miguel Trasobares Baselga sagt:
24. Oktober 2024Favorite aspect: Using an LLM to give characters unique interactions with the player seems very interesting, both because nothing would be 100% scripted and it gives you a wide range of possible options for interaction.
Least favorite aspect: The actual fight mechanics seem a bit forgotten, if the player is gonna spend most of the time fighting with robots I am not quite sure if the conversation mechanic can compensate the core gameplay.
Suggestion: Integrating the conversation mechanic into the game, not only as something cosmetic but with influence in the game would be awesome. Maybe you can add some interaction by talking with your opponent trying to fool him, with the crowd giving you cheers or maybe with the organizers for dropping bonus consumables in the arena (here I am thinking like in "The Hunger Games" were participants get help from outside depending on audience and so on). Maybe you can also consider talking with some AI in the robot to activate perks/weapons/etc.
James Pickrell sagt:
27. Oktober 2024Agree on the importance of the voice mechanics.
Haorui Tan sagt:
24. Oktober 2024Favorite aspect: VR interaction is very interesting
Least favorite aspect: Dialogue might become repetitive over time if the AI responses lack enough variety.
Suggestion: Introducing a system where NPCs' personalities evolve over time based on their interactions with the player would add depth and variety to the dialogue, making each encounter feel more unique.
James Pickrell sagt:
27. Oktober 2024The character dialog is different every time.
We are working to make sure that the character dialog is consistent over time. We dont want them to contradict themselves.
Jialin Yang sagt:
24. Oktober 2024Favorite aspect: Using LLMs to mimic famous characters is a good idea.
Least favorite aspect: Speech-to-text input may not be that reliable. Imagine playing the game with a TV on...
Suggestion: Maybe give more focus on chain reaction? Now it seems that taking advantage of chain reaction isn't in the core of the game.
James Pickrell sagt:
27. Oktober 2024The characters are hilarious.
I hope the rest of the game will be as fun.
Yifei Liu sagt:
24. Oktober 2024Favorite aspect: No one can resist the temptation to pilot a mech.
Expectation: If conditions allow, could a feature be added that enables the mech to briefly fly or glide through a jump? It might not be very useful, but I simply want to experience mech flying in VR.
James Pickrell sagt:
25. Oktober 2024I like it. I will try to make this happen.
James Pickrell sagt:
25. Oktober 2024How about an afterburners button? You press the button and whooosh kaboom you jump over your opponent and put him at disadvantage...
Mahdis Sabzevarzadeh sagt:
25. Oktober 2024Favorite aspect I really like the aspect of creating NPC characters as a mix of historical and fictional figures, I'm very interested to interact with different NPCs and see their dialogs
Least favorite aspect The fighting perspective with use of controller free hand-tracking system doesn't seem very interesting to me for a game whose objective is to fight in an arena
Suggestion I would focus fighting mechanics to be done with a controller/mouse and keyboard. Maybe some other aspects of the game like interaction with NPC or menu can be done by Meta 3
James Pickrell sagt:
27. Oktober 2024The historical character simulations are hilarious.
They are not always accurate on facts and some of what they say is pure hallucination, but it is funny.
Yaxuan Dai sagt:
25. Oktober 2024Favorite aspect: The characters introduced with their own background. This might double my playing time to explore setting of different characters.
Least favorite aspect: I thought this is a VR/AR game according to your selection of device. But I have no idea how could this be corporate with the setting that your robot is giant. Are there any connection?
Suggestion: If the interaction with NPCs is via audio, I would suggest testing with noise disturbance from the first version for the sake of the final demo.
James Pickrell sagt:
27. Oktober 2024So far, it works.
Non-English speaking accent is also a concern.
Our tests have been simple but so far it works.
Cynthia Drews sagt:
25. Oktober 2024Most favorite aspect:
Differently styled NPCs! I really wish they all have different fighting styles too! Or special abilities that they can use like in Overwatch or LoL.
Least favorite aspect:
I can't imagine the barrel aspect having much of an impact yet. I think more complex maps that aren't just planar could help with that. Quake had some cool maps with different floors.
Suggestions:
Will the player be able to channel communication to other players or NPCs, or will it only be broadcast in a specific area? And is it more push-to-talk or activation by voice? I know you check if "Exit the game" is possible in the current context if it's accidentally said in the middle of the game, but it could also be a disadvantage if the AI is always listening. The same goes for gestures. If a friend comes over and starts talking to you while you're showing them the game, and that would interfere with your game, that would be a problem. Or if someone wants to stream their game and entertain an audience.
James Pickrell sagt:
27. Oktober 2024These are good questions on the communication.
I was kind of thinking, in person maybe 10 meters range.
While in robot, the radio has infinite range.
As to when the microphone is on, that is also a good question. We will have to try it and see what works.
I think it is important that the player has control of when he broadcasts or speaks to other players.
So the player could trigger the microphone with one clap for on and one clap for off. Or press a certain button on the controller. Or say "Talk" and "Done talking."
Andrey Roytman sagt:
26. Oktober 2024Most favorite aspect: text-to-speech mechanic.
Least favorite aspect: controller-free robot controls in combat.
James Pickrell sagt:
27. Oktober 2024Using the hands-only control feature is a matter of choice. We will also support Quest controller and mouse. The user can choose.