Friday, November 17, 2017

Space Engineers: Major Physics Overhaul


SUMMARY:
  • Massive physics overhaul
  • Pistons, rotors, landing gears and more
  • Grid deformations - Improved, optimized, tweaked and balanced
  • Optimizations - Less lag, more fun
  • Improved subgrid behavior and fragility
  • Stronger constraints for mechanical blocks so they don’t break so easily
  • Less Clang, more engineering

With today’s major update, 1.185.0, we are releasing a large overhaul to the physics in Space Engineers. The update is primarily focusing on pistons, rotors, landing gears, and grid deformations.



The original vision of Space Engineers was to build a game where physics behave exactly as in the real world. We wanted players to be able to transfer their intuition from the real world to the game world. Things in Space Engineers should behave exactly as one would expect them to behave in real world. There shouldn’t be things that look like they can do something but actually cannot and are there just for visual effect. We wanted to create a game where there are no limits in what players can create.

Just as an illustration: in Space Engineers we have practically infinite worlds, players can seamlessly travel from planets to moons to asteroids, they can build and destroy everything that’s in the world, they can dig through the entire planet which are huge -- up to 120 km, players can build kilometer-long ships while being completely destructible. Ships and stations have their own mechanics - pistons, rotors, thrusters, gravity generator, conveyors, programmable computers and many more.

On top of that, everything is destructible, deformable, recalculated in real time, different blocks have different strength, gravity and other forces push on objects, ships crashes and collisions feel like in real world. We consider these to be the core elements of Space Engineers. 


This has been a top priority for me and the SE team during the last year - to have these things be as robust, stable, and intuitive as possible.

The game’s physics are now more stable and creations shouldn't break, explode or
do uncontrollable things
under normal conditions with default settings. But if players override safe parameters (via mods or in-game sliders) and really push our engine, things may get uncontrollable. However in general, the physics will now remain stable even under much more stressful conditions than before.

In other words, Space Engineers physics are now very stable and robust.

What is the worst case scenario that can still happen in game?

Setting max piston forces to very high values can cause clanging, as well as connecting grids with multiple constraints like a square array of connectors or a long chain of pistons.

I am very proud of what my team has achieved in this major update!

Honestly, I don’t think any other group has done more in the realm of real time large scale interactive physics volumetric simulations. No other game or software project is facing these kind of challenges on this scale.

We are pushing the frontier and defining the limits. It's a hard and painful process, but occasionally there are days like today, where we are happy to present our latest iteration!

Because Space Engineers is still under development, there may be bugs (our testers spent hundreds of hours testing, but this is nothing compared to tens of thousands of hours played by our community immediately after every update). Nevertheless, we will do everything to fix any issue as fast as possible.

Disclaimer: This major update focuses on physics, not multiplayer. Although, the latter has received huge improvements as well. There are still situations in multiplayer where players can push physics to the limit and break the game. Our next focus will be these edge cases.



Teaser of future physics/multiplayer improvements: This video shows the Sacrificial Offering for Almighty Lord Clang in multi-player, with 500ms simulated ping, proper predictions, no cheating.
However, the game should be stable under default conditions, both in single-player and multiplayer.


This is a big day for all space engineers!

Space Engineers is still in development. Everything in the game is subject to change.

Thank you for reading and we look forward to hearing your feedback on this update.

Marek Rosa
CEO, Founder Keen Software House
CEO, CTO, Founder GoodAI


For more news:

Space Engineers: www.SpaceEngineersGame.com
Medieval Engineers: www.MedievalEngineers.com
General AI Challenge: www.General-AI-Challenge.org
AI Roadmap Institute: www.RoadmapInstitute.org
GoodAI: www.GoodAI.com
Keen Software House: www.keenswh.com


Personal bio:

Marek Rosa is the CEO and CTO of GoodAI, a general artificial intelligence R&D company, and the CEO and founder of Keen Software House, an independent game development studio best known for their best-seller Space Engineers (2.4mil+ copies sold). Both companies are based in Prague, Czech Republic. 
Marek has been interested in artificial intelligence since childhood. Marek started his career as a programmer but later transitioned to a leadership role. After the success of the Keen Software House titles, Marek was able to personally fund GoodAI, his new general AI research company building human-level artificial intelligence, with $10mil. 
GoodAI started in January 2014 and has grown to an international team of 20 researchers.

----------



The following technical post goes into great depth about the changes that were made to Space Engineers for this update. It was written up with diagrams and videos by Martin Pavlicek, one of our programmers here at Keen Software House who worked heavily on this physics iteration.

Physics Improvements

by: Martin Pavlicek

Motivation


More stable simulation and better user experience, while keeping the overall simulation as realistic as possible, have always been major goals of programmers in the Space Engineers team. We wanted to reduce (or rather remove) any chance of random explosions, glitching subgrids (which impact their parent grid and cause massive damage), or anything else that could upset players. During the multi-year development, Keen programmers have worked on many iterations to find a solution.


Welding (aka safety locking)


One solution that we tried using to resolve physics issues in general simulation was welding, also known as safety locking but it turned out to be not a sufficient solution. In theory, making one single rigid body out of multiple connected bodies may sound like a good idea, but in practice, it brings more problems than it really solves.

Let's take a look on one of the main cases that welding should solve.
In a following video, we can see interesting ship made out of multiple connected sub-grids. Welding should protect it from breaking up and make sure that sub grids don't disconnect at high speeds. What we get though is not really what we expected


(Note: Continue reading to see “After” version)

At its core, welding is a nasty trick achieved by dynamic swapping of rigid bodies at the core physics simulation level. Unfortunately, this doesn't come for free and introduces many side effects, like performance issues (Keen eye might notice simulation lag in previous video at 22s), simulation flaws like rigid bodies warping through each other when safety lock is enabled/disabled each frame...


... and risk of random disconnections of landing gears, wheels, rotors or pistons every time the grid is welded/unwelded and rigid bodies are hot-swapped ...


… and last but not least, ugly simulation where every rotor, piston or wheel freezes at high speeds.

This led us to conclusion that welding is not the way we should or want to go and decided to remove it entirely.

Max speed limitations


One of the main issues we had to deal with in past was the bullet-through-paper-problem (this is a well-known problem with real time physics simulation).

Limiting the max speed of in-game objects works pretty well for that case but problems come when sub-grids get limited more than base grid due to high angular velocity. See sketch below.


This usually causes sub-grids to disconnect and clang out at high speeds, particularly in space. We solved this issue by strengthening physical constraints and unlocking the subgrids max speed. This allows them to move as fast as needed to keep up with the still limited main/base/central grid.

By further strengthening and stabilizing the physical constraints that keep the sub-grids together, we achieved pretty stable simulation. To demonstrate our solution, lets take a look at the following case study. Combination of rotors, pistons, landing gears and on top of that, each attached fighter is further stressing the connections with enabled inertia dampeners.

 
Before state


 
and after

Stronger constraints


By further strengthening the constraints, we can push them to the limits now, without risk of piston parts flying all over the place like they usually ended up before.

 
Before state

 
and after

Or, when player wants, constraints can now be configured in a way so they break in predictable manner when stressed too much:



Shared inertia tensor


Another issue we had to solve in this iteration were so called “Jiggling grids”. These are creations with very large inertia tensor and/or mass ratios.

Unfortunately, we hit a wall here. This problem is particularly tricky for computers to correctly detect and resolve. After our experience with auto-safety locking, we decided not to opt for an auto-detection system, but rather give you, the players, full control over it. This will allow you to fine-tune your creations exactly to your needs.

That’s why we introduced so called Shared inertia tensor.

If we take a look at the following sketch: G1, G2, and G3 represent grids connected by physics constraints (blue and red lines), those might be for example rotors.


(User can enable shared inertia tensor option on rotor connecting G1 and G2. Then, these two physical bodies will start to share certain physical properties. Purple boxes illustrate that grouping.)

Now, when G3 gets some external impulse, it will start to move and/or rotate. Red constraint will need to compensate G2 for G3’s displacement to keep their intended relative positions. The question is, how much will the G2 succumb and how much will be the G3’s movement slowed down by G2’s resistance.

The problem becomes more and more obvious the bigger the G1 and G3 are compared to the much smaller G2. Since the Havok constraints are solved one by one, not as an entire system as a whole, G2 will succumb to either G1 or G3 each frame, start to oscillate/jiggle between these two as the G1 and G3 will fight over G2. They’ll never move itself to actually find some common compromise and system will never converge to stable simulation/result.

That’s where the shared inertia tensor comes into the play; to either equalize physical properties across all bodies when enabled on every constraint-- as shown in the example below-- or say which grid will be the master and which one should succumb.

Here is in-game video with before and after.

 
(Central body is too “light” compared to satellite wings and they’ll start to fight over it’s position.)

It also helps to stabilize piston chains with “heavy endpoints”, as you can see on this example. Before, the scene didn’t even load without being immediately possessed by Clang:


Now, the scene loads just fine and pistons are even able to move at low speeds without any help of shared tensor. Unfortunately, as the speed goes higher, the pistons will start to behave as a bead chain and problems described above will start to show in a big way. After enabling the shared tensor, everything just works.

 
(Note that second piston chain is not using the shared tensor, yet still behaves naturally. No Clang present here.)

And finally in gravity.

 
(Note that they’ll bend as expected with disabled tensor sharing. It’s all about setting it up the way you want it.)

Just to not render it as some kind of silver bullet solution to every problem, there are some downsides to this, like simulation inaccuracy compared physical laws, and of course, sometimes we want sub-grids to be submissive as much as possible, for example on tank tracks so they don’t “fight” the main body. That’s why it’s not enabled by default and users should use it wisely and only when need, to achieve more stable creations.

Pistons


In previous iterations, pistons were pretty cheesy (when it came to any physical realism). They were also sometimes a pain to use because of their infinite strength and power to penetrate other solid obstacles.


Simulating some kind of suspension or pneumatic piston is not an easy task in real time physics simulation in general. It’s even worse in a world as dynamic as those Space Engineers can offer.

After some tests we decided to not dive into any kind of complicated piston implementation with strength settings and availability to be pushed back when too much pressure is applied on them. Partially because it would be too complicated and time consuming to make it work “right”, partially because it would ruin some creations that rely on pistons with “infinite force” and no pushback.

Thus we decided for compromise that will solve the worst issues players currently face but it will provide the “backwards compatibility” for those who need it at the same time. That is, keyframed piston with limited impulse.

Piston is natively backed by Havok fixed constraint. This is the stiffest constraint we can use now, as opposed to prismatic constraint which is intended for this scenario by Havok and may provide more accurate simulation, but its hold is quite vague under certain scenarios, especially when CoM is far from constraint pivot location.

When a piston is ordered to extend/retract, the constraint is manually positioned each frame and impulse, needed to get the rigid bodies to new desired position is measured. Whenever the impulse starts to grow over certain threshold, specified by player in GUI, the piston advancement is stopped until the constraint catches up again. This usually happens when piston head starts to collide with some other obstacle. It is either light enough so it can be moved away, or it starts to stress the constraint and stops the piston.

 
Before state


 
And after

 

Phantom forces


One of the big topics in the community are the so called phantom forces. The kind of strange behaviour when pasting a grid in space and then it starts to spin or fly away in an uncontrollable manner.

After investigating the issue, and doing some internal tests, we realized that this particular behavior is hard to control in real-time.
Imagine a simple case where there are two pistons connected to the same rigid platforms.

As long as the length of each of these two pistons stays the same as the other one, everything will be fine. The problem comes the moment when one of them changes its length compared to others connected to same rigid body. The inner stresses/tension that will raise inside the system at that moment will result in a fairly complex simulation problem that real-time physics is not able to correctly resolve in any reasonable manner and usually results in a simulation with systems that end up with more energy than they started with in each frame. Usually spinning, flying in different directions and sometimes clanging grids.

In any case, we are not able to differentiate these simulation flaws from normal simulation.

So, when designing your creation, try to avoid “double-connections” like shown on the example above and whenever you encounter the so called phantom forces, double check every connection on your grid and make sure that it doesn’t “fight” with another one. That way, you should never experience these problems.


Rigid body early deactivation and fixing


In order to save the precious CPU time, we decided to early detect and deactivate stationary grids more aggressively and give Havok stronger leads to not simulate these.

As you can see on following profiler dump, just by not simulating stationary rigid bodies that are fixed to either voxel or another station, we can gain massive performance improvements.


(click to zoom the image)

Yellow: Active
Red: Detected stationary
Blue: Force-fixed static



CoM based thrust


I believe that one of the things regular players will really appreciate is CoM-corrected thrust. Before, the attached subgrids would create so called “subgrid drag” which was annoying in space…


... and things-breaking in gravity.


After:





Sub-grid damage


Damage handling caused by subgrids, for example impact damage, was adjusted in a way that it will be ignored between subgrids of the same physical group.
We will trade a little bit of realism for user experience. Nice creations will no longer get destroyed by accident subgrid impact and it will allow users to make even more immersive creations.

This does not apply on damage caused by ship to ship impacts. Only physically connected grids get this resistance.

Displaced sub-grids


Under high angular velocities, subgrids were displacing or entirely disconnecting due to high forces acting upon the construction. This was also fixed in this iteration.

 
Before state

 
After

Always have safety fuse


When everything fails and the worst is about to happen from some unexpected reasons, it's always good to have safety fuse. That's why we implemented safety detach on all "mechanical connection blocks" (That means Rotors, Pistons, Wheels for now). This will protect users from experiencing all those unexpected situations, when subgrid gets insane speed when corrected after being displaced from its intended position and killing everything alive in its way and/or damaging it's own sub-grids, once and for all.

Satellite in following video is example of so called "Jiggling structure". When safety detach is correctly configured, it will end up "only" with some broken blocks and some subgrids flying away, instead of possessed grid, flying out of control.


Grid deformations


Crashing ship to the planet were causing split parts of the grid falling through the terrain to the center of the planet. This was happening also in collisions with other types of physical objects, although in smaller scale. The root cause was that creating or changing shape in physics also discards all the contact points, which means that for the next simulation frame grid can phase through other entities, until it regains the contact points again. Given the deformations were performed each frame, the grid was for most of the time without contact points, thus phasing through other grids. We have added reintegration step after each change of physics shape.
We have also tweaked all the deformations carefully according to the original release of Space Engineers.

No more ship-eating planets!

Miscellaneous


Here comes some examples of long occurring bugs that were solved along during the process.


Piston and Merge block combo was reported by users over and over again.


 
It was fixed and behaves correctly now.

"Sacrificial Offering for Almighty Lord CLANG" is one of those creations made just to stress-test Space Engineers. Some players may be angry that we fixed it (Well, mostly, phantom forces are still there)

 
Before state

 
After

There was interesting bug in pistons.
Center/extending part was just broken from physics perspective. It was usually just missing or would fly away after first contact. Astronauts or even ships were able to fly through.



And that’s it, for this iteration. For all those who made it this far, this is how that beautiful creature from first case study looks and behaves now, after all the improvements made in this iteration.

 
Graceful at all speeds, don’t you think?

---
Best regards,
Martin Pavlicek

38 comments:

  1. Planets won't eat my ships any more?

    My life... it's back!

    You guys R O C K !

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
    Replies
    1. [don't mind me testing Blogger's actions. original message: ]

      I absolutely adore CoM based thrust!
      Btw, what's the skybox used in the videos?

      Delete
  3. I absolutely adore CoM based thrust!

    Btw, what is the skybox used in the videos?

    ReplyDelete
  4. The skybox is a new thing we are cooking... it's not out there yet (not in this update), but we felt we really want to start using it in all our new videos and screens.

    ReplyDelete
  5. First of all, i really want to commend you for this milestone in development, i am sincerely grateful for all the time and dedication you have placed into making this game and sticking all the way to this day and delivering your roadmaps and milestones, congratulations!

    on that regard i have one small question:

    is it still on your roadmap to implement the windmills and the hydrogen engines previously teased? would you give a thought about implementing other features (food, for example), or cannon turrets? ladders? (wink wink)

    once again, thank you! for your hard work and dedication.

    Sincerely

    A fan since ladders existed

    ReplyDelete
  6. Wow, that's pretty impressive!
    1) Have you upgraded Havok version to make this possible?
    2) What do you mean by strengthening the constraints? You mean increasing the number of iteration steps, or only adjusting some constraint parameters?

    According to older GitHub source code, there's not to adjust on constraints itself (solving method, stabilization factor, maximum forces).

    ReplyDelete
  7. Thanks Kingkrieg :)

    Unfortunately I can't comment on planned/not-planned features... :(

    ReplyDelete
  8. some real improvement here, well done, hoping that the game continues to evolve more, but remains stable in both single and MP

    ReplyDelete
  9. Absolutely incredible. I've never used pistons or rotors much at all for fear of problems... now I can actually start having some fun with them.

    ReplyDelete
  10. Great Update, really nice to see the improvements that the team has made. One thing that I am really waiting for is stable Multiplayer that allows 30+ people without server issues. While this is a step in the right direction do you feel the team will ever reach this level in MP?

    ReplyDelete
  11. As a mechanical engineer I can say that was quite a read. Impressive work. I look forward to playing after work tonight.
    Thanks!

    ReplyDelete
  12. Thank you very much Martin Pavlicek!
    Please give this guy a bonus & a vacation!
    😁😁😁

    ReplyDelete
  13. Happy with everything except center of mass corrected thrust. The whole point of playing this game was because I enjoyed the realism, and center of mass corrected thrust is not realistic. Give it back.

    ReplyDelete
    Replies
    1. realistic thrusters mod seems to do the job

      Delete
  14. Replies
    1. When I was kid, I used to like Ninja Turtles, master Splinter, Shredder, and that kind of people.

      Delete
  15. That was an interesting read. I think a lot of people underestimate how hard it is to do physical simulations, much less in real time.

    Every time I read "Clang" I kept thinking about the C compiler.

    ReplyDelete
  16. Marek and Martin, congratulations for the work. Very good job to all ksh team.
    Very good idea to detail in this post all about the work that was done. This article is very good and detailed.
    While the update was not oriented to multiplayer, I can see large improvements in performance on my dedicated server.

    Regards from all people of space engineers latino dedicated server!

    ReplyDelete
  17. thank you marek (and dev team) for this awesome and fun game! thanks for making physics great again. my utmost gratitude from Canada

    ReplyDelete
  18. I love you guys! I love physics and I have se!

    ReplyDelete
  19. I`m at loss for words, Keen, you`ve done it.

    You`ve just proven how good you are as a team and SE as a game.

    It`s simply incredible what you`ve achived.

    ReplyDelete
  20. Well done!

    I had a thought about the double piston phantom force problem. You could implement a cheat where you detect double-linked pistons and force them to be always be the same length, then have that group of pistons "fight it out" amongst themselves to see what that length should be.

    The side effect would be that no amount of tension between pistons would ever break the grid, but it might be a good compromise?

    ReplyDelete
  21. This update is as cool as an update can get! I'm extremely happy about the armor block damage revamp. Finally seems that armor blocks act like armor blocks, not like explosive cardboard. :)

    But RE "The original vision of Space Engineers was to build a game where physics behave exactly as in the real world"... come on! :D Do thrusters in reality effect the center of mass of entities? :D I still remember when I started to play, naturally assuming that thruster placement matters; and I was disappointed to realize that I can just put a thruster on the very end of a long grid, and it will still move in a straight line without rotating at all.

    ReplyDelete
  22. This update is amazing and your team is doing amazing work! Just one issue I have with this is on a lot of my large ships and ships people have on on the workshop, we use rotors to make small block sub grids for extra detail (furniture, bridge screens....). Now that there is no safety lock, most of these sub grids move out of place and become crooked or just break in general while flying. Please fix :(

    ReplyDelete
    Replies
    1. Hey,

      as you said: there is no safety lock option, so you need to set it up (torque, angle limits) correctly.

      Please send your blueprints to ondrej.nahalka@keenswh.com and I will check it out.

      Ondrej
      KeenSWH

      Delete
    2. Hey,

      as you said: there is no safety lock option, so you need to set it up (torque, angle limits) correctly.

      Please send your blueprints to ondrej.nahalka@keenswh.com and I will check it out.

      Ondrej
      KeenSWH

      Delete
    3. Thanks for the tip! I'm not that good with rotors but by adjusting the Torque and angle limits I was able to keep the sub grids in place without movement while flying. I guess I just didn't know what settings to fix. I appreciate the fact that you found the time to offer help, you don't see developers personally offer help to players often....Keep up the good work guys your game is amazing and only getting better!

      Delete
  23. please add safety lock a lot of peoples work is being messed up because of removal

    ReplyDelete
  24. this is truly a milestone! I still kind of hope that SE will at least in name preserve their holy god Clang.
    I'm wondering about the CoM 'correction'. i know this is how engines should work in SE, but CoM and thrust direction and position do play a role in IRL physics. There are mods on the workshop currently, such as "Realistic Thrusters", that are used to increase realism in that area. would you think about adding something along the lines of an arcade/realism mode for physics like you did for sound?
    This was an amazing update and will certainly bring me and perhaps several others back into SE, who before were facing physics and lag issues. Thank you for your work on and for SE (and GoodAI)!

    ReplyDelete
  25. This ^^^^^, many of my creations require it

    ReplyDelete
  26. Do rotors still have the same max rotation speed?
    Example, if you have two rotors, with rotor 2 attached to the rotor of rotor 1, and you set them both to max speed, what will be the net rotational speed of the 2nd rotor?

    ReplyDelete
  27. Please test cars in multiplayer, they twitch and any player nearby gets teleported around. Also with attached thrsuster it gets worse, if you turn them of it gets better.

    ReplyDelete