Monday, May 18, 2015

General AI team at Keen Software House hits 2nd milestone

Today I’m excited to tell you that our general AI project has reached another important milestone.

A quick reminder of what our AI brain team has achieved so far:
  • an AI that can play Pong, a Breakout game (left/right movement, responding to visual input, achieving a simple goal)
  • Brain Simulator (a visual editor for designing the architecture of artificial brains)

The new milestone is that our general AI is now able to play a game that requires it to complete a series of actions in order to reach a final goal. This means that our AI is capable of working with a delayed reward and that it is able to create a hierarchy of goals.

Without any prior knowledge of the rules of the game, the AI was motivated to move its body through a maze-like map and learn the rules of the game. The agent behaves according to the principles of reinforcement learning - this means that it seeks reward and avoids punishment. It moves to the place in the maze where it receives the highest reward, and avoids places where it won’t be rewarded. We have visualized this as a 2D map, but in fact the agent works on an arbitrary dimension and the 2D map is our visualization only. The agent actually "sees" 8 numbers (8-dimensional state space) which change according to the agent’s behavior, and it must learn to understand the effects of its actions on these numbers.

Here you can see an example map of the reward areas - the red places represent the highest reward for the AI, and the blue places represent the least reward. The AI agent always tries to move to the reddest place on the map.

Visualization of the agent’s knowledge for a particular task, which changes the state of the lights. It tells the agent what to do in order to change the state of the lights in all known states of the world. The heat map corresponds to the expected utility (“usefulness”) of the best action learned in a given state. A graphical representation of the best action is shown at each position on the map.

The agent's current goal is to go towards the light switch and turn on the lights.

The maze we are using is one where doors can be opened and closed according to a switch, and lights can be turned on or off according to a different switch. When all of the doors are open, the AI agent moves easily through the maze to reach a final destination. This kind of task only requires that the agent complete one simple goal.

The agent uses its learned knowledge to reach the light switch and press the button in order to turn on the lights.

However, imagine that the agent wants to turn on the light but the doors to the light switch are closed. In order to get to the light switch, it first has to open the door by pressing a door switch. Now imagine that this door switch is located in a completely different part of the maze. Before the AI agent can reach its final destination, it has to understand that it cannot move directly to its goal location. If first has to move away from the light switch in order to press a different switch that will open the necessary door.

Our AI is able to follow a complex chain of strategy in order to complete its main goal. It can assign a hierarchical order to its various goals and plan ahead so it reaches an even bigger goal.

The agent solves a more complex task. It has to open two doors in a particular sequence in order to turn on/off the lights. Everything is learned autonomously online.

How this is different from Pong/Breakout, our first milestone with the AI

The AI is able to perform more complex directional tasks and (in some ways) in a more complex environment. While in the Pong environment it could only move left or right, in this maze the agent is able to move left/right, up/down, stay still, or press a switch.

Also, the AI agent in Pong acts according to visual input (pixels), which is raw and unstructured information. This means that the AI began to learn and acted according to what it could "see." In the maze, the AI agent has full and structured information about the environment from the beginning.

Our next step is to have the AI agent get through the maze according to visual, unstructured input. This means that as it interacts with its environment, it will build a map of the environment based exclusively on the raw visual input it receives. It won’t have that  information about the environment when it starts.


How the algorithm works

The brain we have implemented for this milestone is based on combination of a hierarchical Q-learning algorithm and a motivation model which is able to switch between different strategies in order to reach a complex goal. The Q-learning algorithm is more specifically known as HARM, or Hierarchical Action Reinforcement Motivation system.

In a nutshell, the Q-learning algorithm (HARM) is able to spread a reward given in a specific state (e.g. the agent reaching a position on the map) to the surrounding space so the brain can take proper action by climbing steepest gradient of the Q function. However, if the goal state is far away from the current state, it might take a long time to build a strategy that will lead to that goal state. Also, the number of variables in the environment can lead to extremely long routes through the "state space", rendering the problem almost unsolvable.

There are several ideas that can improve the overall performance of the algorithm. First, we made the agent reward itself for any successful change to the environment. The motivation value can be assigned to each variable change so the agent is constantly motivated to change its surroundings.

Second, the brain can develop a set of abstract actions assigned to any type of change that is possible (e.g. changing the state of a door) and can build an underlying strategy for how this change can be made. With such knowledge, the whole hierarchy of Q functions can be created. Third, in order to lower the complexity of the problem, the brain can analyze its "experience buffer" from the past and eventually drop variables that are not affected by its actions or are not necessary for the current goal (i.e. strategy to fulfill the goal).

A mixture of these improvements creates a hierarchical decision model that is built during the exploration phase of learning (the agent is left to randomly explore the environment). After a sufficient amount of knowledge is gathered, we can "order" the agent to fulfill a goal by manually raising motivation value for a selected variable. The agent then will execute the learned abstract action (strategy) by traversing the strategy tree and unrolling it into a chain of primitive actions that lie at the bottom.


Our motivation

Like with the brain’s ability to play Pong/Breakout, this milestone doesn’t mean that our AI is useful to people or businesses at this stage. It does mean that our team is on the right track in its general AI research and development. We’re hitting the milestones we need to hit.

We never lose sight of our long term goal, which is to build a brain that can think, learn, and interact in the world like a human would. We want to create an agent which can be flexible in a changeable environment, just like human beings can. We also know that general AI will eventually bring amazing things to the world – cures for diseases, inventing things for people that would take much longer to invent without the cooperation of AI robots, and teaching us much more than we currently know about the universe.

---

Thanks for reading!

If you’d like to see future updates on the general AI project, you can follow me on Twitter http://twitter.com/#!/marek_rosa or keep checking my blog: http://blog.marekrosa.org

Thursday, May 14, 2015

Space Engineers – full source code access, total modifications and 100,000 USD fund

Today we have a very important announcement for our modders and our community. We decided to give you 100% complete access to Space Engineers’ source code. This comes as a continuation of our decision to give more freedom to modders and community.

Please do not confuse this with “open source” or “free software”: While we are opening the possibility to read and alter Space Engineers and VRAGE source code, we’re not making Space Engineers or VRAGE free.

I have spoken about ideas like this in my previous blog post - http://blog.marekrosa.org/2014/09/programming-is-coming-to-space-engineers_15.html but back then we were assuming we would open only some parts of the game. Later we came to the conclusion that opening the entire code base will be much better for everyone.

Modders can now develop the same way Keen Software House developers do (this however does not mean we will stop working on the game).

During the last few months we have been removing parts of the obfuscation on the Space Engineers executable and libraries. After that we  uploaded the source code of Space Engineers and VRAGE engine on GitHub as a private repository, let selected modders look at it and provide us with  feedback, and today we’re making the repository public.

There are very few game studios that have ever done this, probably none during their early access period. It’s an experiment that someone has to try, and the chances that it will be beneficial for the future of Space Engineers is high and worth taking the risk.

Pros:
  • Total conversions are now possible
  • More and better content created by modders
  • Allowing people to have more fun with our game by letting them modify it while we keep adding core features and updates just like we did for the last 2 years
  • Modders can make server-side mods easily
  • Modders can contribute to Space Engineer’s core development if they want
  • Modders can extend the official ModAPI interface and we will merge their changes with the core game

Cons:
  • “Competitors” can steal our ideas and algorithms. Our answer: if it is so easy to clone our games and development methods, then we probably aren’t doing anything special. We believe that our competitive advantage is in being leaders, not followers.
  • Hackers will have an easier time cheating in multi-player. Our answer: this is true, but with new multiplayer, the server will have authority and will validate everything sent by clients. If our implementation has flaws, open access to our source code and the visibility of potential flaws will push us to fix these cases faster.
  • We can be accused of going open source and abandoning the development of Space Engineers. Our answer is definitely NO. As we stated above, this is mostly for giving complete freedom to our modders while we continue the game’s development without any changes (through weekly updates and keeping our development plan as described in our previous blog post). Space Engineers is still selling very well and only a crazy person would abandon the game! We just want to give people the chance to modify all aspects of our game and experiment with it while we keep doing what we are doing.

In the worst case scenario, we will revert back to obfuscating the source code and with our frequent update routine the non-obfuscated source code will soon become old and obsolete.

NOTE: we are organizing an IAmA on Reddit today at 20:00 CET / 14:00 EST where you can ask us more questions in case something hasn’t been explained clearly enough (you can read the IAmA here: http://www.reddit.com/r/IAmA/comments/35yv9f/we_are_keen_software_house_developers_of_space/).

How it works?


Go to GitHub, download and unpack the source codes, open it in Visual Studio Community Edition (free), modify the source code, launch the game! See this video guide:



This new type of mods can’t be uploaded on Steam Workshop (unlike ModAPI mods) because of security risks. We are assuming that modders will find ways to share their mods with players. Server side mods will be installed on servers by server administrators, so it’s not necessary to distribute them to clients.

Modders will be able to contribute to ModAPI in vanilla game. It often happens that modders want to create a  mod, but they are unable to do it because ModAPI has a limited interface. In this case, they will simply add support for it (usually it means creating a small public interface for an existing feature). We will review their changes and when it meets quality and security requirements, we’ll merge it with the core game.

General discussion, suggestions and questions can be posted on the dedicated Source Code sub-forum: http://forum.keenswh.com/forums/source-code.423135/

What kind of mods can be made?


Workshop mods: Nothing changes for workshop mods, except modders who wants to get involved in ModAPI improvement now have a chance.

Total conversion mods: These mods can ignore the limits of ModAPI and modify anything in the game. Because we’re unable to guarantee security, these mods can’t be uploaded to Steam Workshop and must be distributed in a different way. Games like Counter-Strike, DOTA and DayZ were created in similar fashion.

When an author of a total conversion mod wants to release his mod on Steam (as a standalone game or Space Engineers DLC/mod) we’d be happy to help him with that. Contact us: info@keenswh.com

Standalone games: if you decide that modding is not for you and you want to build a brand new game based on Space Engineers’ and/or VRAGE source code and assets, don’t hesitate to contact us: info@keenswh.com


What are modders saying?


Digi: “This will open the possibility for a lot of extra improvements to the game and its modding API, from the fans that want to get involved.

Digi is the author of several very popular mods, including Helmet 3D hud and Gravity collector. He also contributed to ModAPI improvement. His workshop: http://steamcommunity.com/id/hunterdigi/myworkshopfiles/?appid=244850

Tyrsis: “I personally would love to modify some things, as I'm limited to responding to events / sending sync packets in order to make things happen on the client. Having more control at the source level would be interesting.”

Tyrsis is author of SEServerExtender and very popular mods including Automated inventory sorting and GPS and waypoints. His workshop: http://steamcommunity.com/profiles/76561198023356762/myworkshopfiles/?appid=244850

Malware: “I'm personally very happy to be granted access to the source code, but I'm a bit worried what will happen if you release it to everybody.

Malware is a very active member on GitHub, he already contributed by adding several features to ModAPI. His workshop: http://steamcommunity.com/id/mal-ware/myworkshopfiles/?appid=244850


$100,000 fund


To support our modding community even more, we reserved $100,000 of our own money as a fund for modders who want to build total conversion mods based on Space Engineers or VRAGE engine.

We’re considering several ways how this fund could work: grants with no strings attached (see how Epic does it with UDK – https://www.unrealengine.com/unrealdevgrants), zero interest loans and Kickstarter support. We can provide PR, marketing and promotion on our social media.

Why are we doing this? We believe that our VRAGE engine is unique in certain aspects and it would be great to see what other people can create with it. VRAGE supports fully dynamic and volumetric environments where players can build and destroy everything around; our game world is unlimited in size and can be procedurally generated.
More info about VRAGE: http://www.keenswh.com/vrage.html

Who are we looking for? Anyone who’s interested in our technology and wants to create game or total conversion using VRAGE engine or Space Engineers. We’re looking for people who are enthusiastic about the same things as we are; game development, sandbox games, destructible environments and unlimited gameplay experience.


Legal notes



If you don’t want to read the lawyer mumbo jumbo, please just keep in mind the following:
  • The source code and art assets must not to be mistaken for free software, an open source in a free-software activist understanding, copy-left or public domain software. All source code and art assets remain copyrighted and licensed by KEEN SWH LTD. and you are allowed to use them (modify, tweak, make a derivative work, distribute, etc.) only under following conditions.
  • The source code, modifications or derivative works can be distributed only if they are intended to be used as a mod for Space Engineers, and only if valid customers would be able to use them. You are not allowed to bypass this restriction and create a standalone application or use our code in your projects.
  • In other words, everyone who wants to develop and/or use mods based on our source code is required to own a valid Steam version of Space Engineers. You are not allowed to use our source code in an application other than Space Engineers. You are not allowed to remove the Steam integration from the code.
  • You are not allowed to distribute original art assets (textures, models, fonts, etc.) except if you won’t be able to make your modifications without our original files (e.g. painting a mustache on our astronaut texture and then distributing it as a mod to Space Engineers is allowed)
  • You are allowed to share the source files with other developers (crediting us when logical/reasonable), downloading and compiling the game locally, modifying and tweaking the game locally, and even publicly forking, modifying, and tweaking the game source on GitHub.
  • The origin of this software must not be misrepresented; you must not claim that you wrote the original software.
  • Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software. This notice may not be removed or altered from any source distribution.
  • By making a pull request on our GitHub repository, you're stating that you're the author of these changes or have rights to the changes you've made and you're giving us the right to use it in any way.
  • Commercial usage is allowed only after you obtain an agreement from us: info@keenswh.com
  • We reserve the right to change this license at any time with or without notice, with immediate and/or retroactive effect. We believe that what we ask is reasonable, so please don't try to bypass it. We're trying to be open and honest, and we hope people treat us the same way back.
  • Don’t be afraid to experiment with the source code. If you break something, you can always revert by downloading it again.
  • If there's anything legal you're wondering about that wasn’t answered: don't do it or ask us about it. If there's anything you don't understand or you consider confusing, please email us your suggestions: info@keenswh.com
  • That's it! No more rules. If the above wasn't clear, here's a summary: use this source code only for developing mods for Space Engineers.


Conclusion


I would like to end by saying again that there is no need to fear that we are abandoning Space Engineers’ development. We are very concentrated on the game’s development and on the players’ suggestions. We assure you that we will continue at the same pace as we have done for 2 years and deliver a game that all our players will love playing! If you see someone who doesn’t fully understand our arguments, please refer them to this blog post. Thank you!

In case you have more questions or if something hasn’t been explained clearly enough, we are organizing an IAmA on Reddit today at 20:00 CET / 14:00 EST (you can read the IAmA here: http://www.reddit.com/r/IAmA/comments/35yv9f/we_are_keen_software_house_developers_of_space/). Also, please don’t hesitate to ask your questions in the comment section below. We may compile a FAQ later.

SPECIAL NOTE: we noticed that many of you are eager to see planets, so this is a small preview from the development (seamless transition, 100km+ in size, procedurally generated, deformable/destructible, almost infinite amount, gravity and atmosphere... still waiting for trees, grass, etc.) :)






---

Thank you for reading this!

Please keep posting your feedback and suggestions under this blog post or our forum http://forum.keenswh.com/forums/source-code.423135/. We can’t reply to every comment that you send, but I can assure you that we read as many as possible.

Thanks!
Marek Rosa

---

For the latest news on our games, follow us on Facebook or on Twitter.

Space Engineers on Facebook: https://www.facebook.com/SpaceEngineers
Space Engineers on Twitter: https://twitter.com/SpaceEngineersG
Medieval Engineers on Facebook: https://www.facebook.com/MedievalEngineers
Medieval Engineers on Twitter: https://twitter.com/MedievalEng

Tuesday, May 5, 2015

Medieval Engineers – Multiplayer & development update 5/5/2015

This is a more detailed summary of the multiplayer update that we just released in Medieval Engineers and an insight of what are we working on right now and what you should expect in the near future.

The multiplayer for Medieval Engineers has been just released. We have been working on it for the last few weeks and we believe that it is in a good shape – it works in creative mode and in survival mode. We are also expecting that the player base is going to increase significantly – this is what our experience tells us when multiplayer was added in Space Engineers (which got only creative mode multiplayer on its first iteration).


I want to clarify that this is still not the new multiplayer layer (based on Raknet) that we have been working on lately (Ondrej has described it in the previous blog post), but the same version of multiplayer that’s used in Space Engineers. We still need more time to work on this new version. However we thought that it is a good idea to add multiplayer to Medieval Engineers now since it’s more fun when you play with your friends and also because many of you have been asking for it for a long time! It supports all the features that are now in the game, in both Creative and Survival modes, building and destruction, structural integrity, mechanical parts, blueprints, mods, synchronized daytime and AI characters (barbarians, peasants and deer). In more simple words, you can do everything that is possible now in the single-player but with your friends.

The game’s development is progressing in a good pace. We have implemented enough blocks already in the game so you can build any kind of creations you desire. Our programmers are focusing more on the survival mode and you should expect more survival based features in the upcoming weeks. Some of the most important are a battle system, melee fight, etc… Stay tuned!

Lastly, our team keeps expanding. We have some more programmers starting this week and more are expected to come during this month (we still have various open positions in case you or any of your friends are interested in joining our team - http://www.keenswh.com/jobs.html). 

We are also planning to move (again :-)) into a new and much bigger office during the summer.

---

Thank you for reading this!

Please keep posting your feedback and suggestions to our FB page https://www.facebook.com/MedievalEngineers or our forum http://forum.keenswh.com/forums/suggestions.413446/. We can’t reply to every comment that you send, but I can assure you that we read as many as possible.

Thanks!
Marek Rosa

---

For the latest news on our games, follow us on Facebook or on Twitter.

Medieval Engineers on Facebook: https://www.facebook.com/MedievalEngineers
Medieval Engineers on Twitter: https://twitter.com/MedievalEng
Space Engineers on Facebook: https://www.facebook.com/SpaceEngineers
Space Engineers on Twitter: https://twitter.com/SpaceEngineersG