FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Throwing in some ideas

 
Post new topic   Reply to topic     Forum Index -> Yage
View previous topic :: View next topic  
Author Message
jdrewsen



Joined: 05 Mar 2008
Posts: 21
Location: Copenhagen, Denmark

PostPosted: Thu Dec 23, 2010 9:02 am    Post subject: Throwing in some ideas Reply with quote

Hi,

I've been digging a bit into the src code. I really like what I've seen so far!

Of course there are room for improvements and here are some that I would like to discuss:

Serialization
The best way to test out the features and is to setup different scenes
and play around with it. Currently this means that you have to code
a new scene, enable it in the build process and run it.

I think that reading scenes from files and just as important writing
scenes is crucial. It makes the round trip times for different setups
much shorter e.g. when you're doing debugging as well.

Whether to use a textual/binary and new/existing format is up for discussion.
If we do I right we can support all kinds - starting out with a single one of course Smile

Binary builds
The best way to get excited about a piece of software is if you can actually try i out.
I think that creating binary executables regularly that are able to run the serialized files
and allow people to change a bit in the scene files would give more attention to the
project. On a personally level it would also be a great way to show friends what I'm
fooling around with in my spare time.

Node classes
I see some issues ahead regarding the design around different Node classes. I've previously made almost exactly the same design and got into these problems.

Currently most kind of scene content is derived from Node which contains pos,rot,scale and velocity.
When you want to get some geometry to the scene you add the Model node derived from Node.

Now this is how I made the wrong design choice: A new node CollidableModel was added (derived from Model) because I needed support for collisions. Soon after I needed support for rigidbody dynamics and "naturally" derived this from CollidableModel as RigidBodyModel.

Now let's say you have a NPC e.g. a soldier. For this a RigidBodyModel is probably not appropriate since humans in games often needs to move not according to the law of physics e.g. quick sidestep/backstep. This is much easier to do without applying forces to a rigidbody. When the soldier dies it would be neat if he could fall naturally using rigidbody dynamics.

For this to work there has to be some way to "convert" from the soliers CollidableModel/Model into RigidBodyModel which means changing class type.

A similar thing can happen when an character dies. Often you want to let the actor stay around dead on the ground for a while and fade him out of the scene. In this situation you probably want ignore collisions with the character which means that the class has changed from e.g. CollidableModel to Model.

The point here is that a character often needs the characteristics of all the classes just at different times during the game. This suggests that the features should not be encoded into a class tree but be toggleable.

Components
A way several other engines has overcome this is to use components instead. The node tree in this case simply a way to specify a hierarchy of node for transformations. Features of the nodes e.g. lighting, geometry, collidable etc. is specified by attaching there features as components to a node. This way you can toggle a feature by adding/removing it from the node.

In the example of the solider he would start out being a Node with a Collision component attached. When he dies a rigidbody component could be attached. When he fades away the rigidbody+collision component is removed a maybe a fade component is attached.

You could also do this by creating a "super" Node that contains all the features and just enable/disable at will. Unfortunately this enforces all Node to be quite heavy weight.

Some engines also allow scripting the behavior as a component of the Node.

Scripting
Scripting is a very convenient to have in a game and is already on the roadmap I can see Smile

I don't know if this is far fetched but... what about using D as the scripting language. Embed the compiler and compile "script" D files into a dynamic library that the engine can load at runtime. The D compiler is fast enough to do this I think. It would diffinitely by the fastest gaming scripting out there. Or what... to crazy?

Data oriented
Currently all nodes are positionable and I bet we could speed up the update() a greate deal by making the pos,scale,orietation data oriented ie. Put all of it into a large consecutive array aligned at cache lines in order to also utilize several processors. I guess the same could be done for other parts as well?

D2 migration
D2 has a lot of new goodies (especially regarding multicore) and derelicht2 is usable AFAIK. Unfortunately tango does not seem to do much in order to be usable with D2.
Mayby porting the tango dependant parts of yage to phobos is a viable way to go? This would also remove a dependency.

anyway..

I hope I'm not rambling too much here - but I would very much like to know your thoughts on all this.

And a merry christmas!
/Jonas
Back to top
View user's profile Send private message
JoeCoder



Joined: 29 Oct 2005
Posts: 294

PostPosted: Thu Dec 23, 2010 11:28 am    Post subject: Reply with quote

Thanks for your analysis. I enjoy seeing people dig into Yage.

Serialization

Have you looked at the orange project here on dsource? Would xml or json be a good choice for a serialization format? Should serialization wait for the scene graph to become more finalized?

Binary builds

Every few months when things are looking reasonably stable, I had been committing a yage3d.exe in the bin folder. It's bad svn policy, but it lets someone try it out without a compiler (at least on windows).

If you want to make some demos, I'll host them on yage3d.net and add them to the downloads page.

Node Classes

I had been thinking about this in regard to physics as well. D doesn't have multiple inheritance. Mixins could work, but can get messy?

For your situation, why not make a CollisionNode as a child of Node, and it can be assigned a Geometry to act as its collision hull (or have options for cube/spherical/plane collision). Then add a ModelNode to this as a child in the scene graph. You could also make a RagDollModel that extends ModelNode and updates the Geometry itself. Ragdoll physics could be enabled/disabled. This Geometry instance is assigned to the CollisionNode, so no matter what mode it's in, it still behaves correctly.

Components
Ogre does this component approach (Nodes and Entities, it calls them) and I didn't care for it there. I think it's possible to do all of this with the normal scene graph by attaching lights and geometry to a collision node, and it keeps the API simpler.

I agree that a "super" node would be too heavy.

Scripting

I think the main purpose of scirpting languages is to 1. Try something quickly without waiting several minutes for all of your c++ code to compile. 2. Provide a language that's much easier to use than C++. D already does both of these quite well, which decreases the need for Yage to have a scripting languge at all.

However, it would be very nice to open up a command prompt inside a yage-powered game and start typing code. This DOES decrease development time because then you don't have to stop and reload your game after every change.

At one point there was a project on dsource (ddl?) to load obj files at runtime, but they had a lot of trouble making this work in a cross-platform way and eventually it was abandoned? There was also the miniD language which was very similar to D and very performant, but also abandonned.

Some have suggested Lua, but I don't think I could ever get used to arrays being 1-indexed.

Python is widely known, but slow and rather different in syntax from D.

Google's V8 javascript engine is very fast, bsd licensed, almost everyone knows javascript, so this has become my first choice.

Data oriented

This exactly! Take a look at tests/benchmark/culling.d where I have already been investigating this. However, there are other optimizations I want to explore on the scene graph before trying this on a big scale.

D2 migration

Don't forget interfacing with C++ libraries! Bullet is an amazing c++ physics engine.

I think the longer I wait, the more stable D2 will become. Yage migrated from phobos to tango a couple years ago. Going back to phobos1 or 2 would hurt collada loading since neither have decent or fast xml parsers. Also, I'm not sure if D2 prints a stack trace on failure. Many of the debuggers have been unstable for me and this has been a big time saver.

Yes, I definitely want to go to D2+Phobos2, but not yet.
Back to top
View user's profile Send private message
Trass3r



Joined: 29 Feb 2008
Posts: 66
Location: Germany

PostPosted: Fri Dec 24, 2010 7:47 am    Post subject: Reply with quote

JoeCoder wrote:
At one point there was a project on dsource (ddl?) to load obj files at runtime, but they had a lot of trouble making this work in a cross-platform way and eventually it was abandoned? There was also the miniD language which was very similar to D and very performant, but also abandonned.

I think nobody works on DDL at the moment but h3r3tic has a new linker for it in some other private repository.

Quote:
Some have suggested Lua, but I don't think I could ever get used to arrays being 1-indexed.

Though LuaJIT is extremely fast Smile

Quote:
Don't forget interfacing with C++ libraries! Bullet is an amazing c++ physics engine.

It's still limited. I guess you need to use something like SWIG for such a big library, it just got D support.

Quote:
I think the longer I wait, the more stable D2 will become. Yage migrated from phobos to tango a couple years ago. Going back to phobos1 or 2 would hurt collada loading since neither have decent or fast xml parsers. Also, I'm not sure if D2 prints a stack trace on failure. Many of the debuggers have been unstable for me and this has been a big time saver.

Yes, I definitely want to go to D2+Phobos2, but not yet.

I think stacktraces are already implemented for Linux and there's a module for Windows too: http://3d.benjamin-thaut.de/?p=15
Back to top
View user's profile Send private message
jdrewsen



Joined: 05 Mar 2008
Posts: 21
Location: Copenhagen, Denmark

PostPosted: Mon Dec 27, 2010 5:27 pm    Post subject: Reply with quote

JoeCoder wrote:
Thanks for your analysis. I enjoy seeing people dig into Yage.

Serialization

Have you looked at the orange project here on dsource? Would xml or json be a good choice for a serialization format? Should serialization wait for the scene graph to become more finalized?


I've know of its existence. I just took a closer look at it now. My main concern is that it seems pretty tied to the internal data model. It is kind of structured memory dump in some way. I does seem pretty easy to use though.

It would be nice to handle network serialization, file serialization and serialization of animateable properties using one system. Or maybe this would be over-engineering at this point?

JoeCoder wrote:

Binary builds

Every few months when things are looking reasonably stable, I had been committing a yage3d.exe in the bin folder. It's bad svn policy, but it lets someone try it out without a compiler (at least on windows).

If you want to make some demos, I'll host them on yage3d.net and add them to the downloads page.


Great! As soon as a scene serialization is in place I think it will make sense and I'll do that.

JoeCoder wrote:

Node Classes

I had been thinking about this in regard to physics as well. D doesn't have multiple inheritance. Mixins could work, but can get messy?


I think mixins are nice for some problems... I don't know if this is such situation though.

JoeCoder wrote:

For your situation, why not make a CollisionNode as a child of Node, and it can be assigned a Geometry to act as its collision hull (or have options for cube/spherical/plane collision). Then add a ModelNode to this as a child in the scene graph. You could also make a RagDollModel that extends ModelNode and updates the Geometry itself. Ragdoll physics could be enabled/disabled. This Geometry instance is assigned to the CollisionNode, so no matter what mode it's in, it still behaves correctly.


How would the CollisionNode "communicate" to the RagDollModel where collisions (contact points) has occurred ( in order to do collision responses)? Wouldn't it mean that the CollisionNode must know which RagDollModel to handle... or the RagDollModel must know which CollisionNode is should get collision info from? This could be done implicitely by making it a rule that the ragdolls get collision info from the nearest CollisionNode or something like that - but that is kind of magic Smile

Another problem is how the RagDollModel would "communicate" the changes in position, velocity, orientation to the parent CollisionNode since it is now "controlling" its parent node ie. the normal "parent moves child" pattern is reversed in this scenario.

I would really like to keep all stuff in the node tree as well - but I think it is hard to get as clear a design as with the component design.

Any ideas of how to do the collision response and position communication smoothly?

JoeCoder wrote:

Components
Ogre does this component approach (Nodes and Entities, it calls them) and I didn't care for it there. I think it's possible to do all of this with the normal scene graph by attaching lights and geometry to a collision node, and it keeps the API simpler.

I agree that a "super" node would be too heavy.


If the collision info response above can be solved elegantly I think same solution can make components unimportant yes Smile
Another concern could be that all things attached to a node contains position,rotation,velocity etc. even though it is not used. This creates some memory overhead. But maybe it's better to wait and see if this becomes a problem.


JoeCoder wrote:

Scripting

I think the main purpose of scirpting languages is to 1. Try something quickly without waiting several minutes for all of your c++ code to compile. 2. Provide a language that's much easier to use than C++. D already does both of these quite well, which decreases the need for Yage to have a scripting languge at all.

However, it would be very nice to open up a command prompt inside a yage-powered game and start typing code. This DOES decrease development time because then you don't have to stop and reload your game after every change.


A cool thing could be if yage was detecting changes to the custom D "script" and compiling into a loadable library while the engine is running and then reloading the library.
This way you wouldn't have to reload your game on each change. This would demand a strict API against the yage engine. I think it is doable but of course ambitious Smile

For a command prompt it may not be the best solution though.

JoeCoder wrote:

At one point there was a project on dsource (ddl?) to load obj files at runtime, but they had a lot of trouble making this work in a cross-platform way and eventually it was abandoned? There was also the miniD language which was very similar to D and very performant, but also abandonned.


yeah ddl seems abandoned Sad


JoeCoder wrote:

Some have suggested Lua, but I don't think I could ever get used to arrays being 1-indexed.

Python is widely known, but slow and rather different in syntax from D.

Google's V8 javascript engine is very fast, bsd licensed, almost everyone knows javascript, so this has become my first choice.


I used python for scripting in a C++ engine I did. You do not want to run to complex code in that language such as AI/steering etc. It is simply too slow unless you just use it to setup stuff for the core engine to perform and other light weight tasks.

I think V8 is pretty nice. Personally I like statically typed languages better e.g. mono and c#? Its very fast as well.
But you've got a point that everyone knows javascript.


JoeCoder wrote:

Data oriented

This exactly! Take a look at tests/benchmark/culling.d where I have already been investigating this. However, there are other optimizations I want to explore on the scene graph before trying this on a big scale.


Cool! I'll have a look.

JoeCoder wrote:

D2 migration

Don't forget interfacing with C++ libraries! Bullet is an amazing c++ physics engine.

I think the longer I wait, the more stable D2 will become. Yage migrated from phobos to tango a couple years ago. Going back to phobos1 or 2 would hurt collada loading since neither have decent or fast xml parsers. Also, I'm not sure if D2 prints a stack trace on failure. Many of the debuggers have been unstable for me and this has been a big time saver.

Yes, I definitely want to go to D2+Phobos2, but not yet.


I'm all for Bullet. I've used ODE myself but it seems that Bullet is more alive and maintained than ODE.


I think I'll look more into the serialization part and see if I can come up with something sane.
Back to top
View user's profile Send private message
jdrewsen



Joined: 05 Mar 2008
Posts: 21
Location: Copenhagen, Denmark

PostPosted: Mon Dec 27, 2010 5:32 pm    Post subject: Reply with quote

By the way. You mentioned integrating Bullet. What are you thoughts on using c++/c dependencies.

Should they be listed in the build instructions as a dependencies or should the source be included in yage and build?
Back to top
View user's profile Send private message
JoeCoder



Joined: 29 Oct 2005
Posts: 294

PostPosted: Mon Dec 27, 2010 5:39 pm    Post subject: Reply with quote

I think it's better to list them in the build instructions as dependencies. This way it's less for us to maintain and we can focus our efforts on yage code. I'm also quite ignorant when it comes to easily building C code across multiple platforms.

On Windows I've just been including the precompiled dll's in the repository, and so far everything I've used has been standard on linux.
Back to top
View user's profile Send private message
JoeCoder



Joined: 29 Oct 2005
Posts: 294

PostPosted: Mon Dec 27, 2010 5:43 pm    Post subject: Reply with quote

Quote:
It would be nice to handle network serialization, file serialization and serialization of animateable properties using one system. Or maybe this would be over-engineering at this point?


I'm not sure either way. Sure they'll be different formats, but a strong reflection class would be helpful to start either. We would send a lot less data across the wire than we would to a serialized dump.

I'll try to respond to your other notes later today.
Back to top
View user's profile Send private message
JoeCoder



Joined: 29 Oct 2005
Posts: 294

PostPosted: Mon Dec 27, 2010 11:56 pm    Post subject: Reply with quote

Components
Quote:
How would the CollisionNode "communicate" to the RagDollModel where collisions (contact points) has occurred ( in order to do collision responses)?

I was thinking of something like this. The CollisionNode and the child ModelNode would be assigned the same RagDollModel geometry:
Code:
class CollisionNode : Node
{   Geometry shape;
   void collideAndDeform(); // simplified
}

auto collisionNode = new CollisionNode(scene);
auto modelNode = new ModelNode(collisionNode);

auto human = new Model("Human.dae");
modelNode.model = human;
collisionNode.shape = human;

human.dance(); // animates model, updates collision hull
collisionNode.collideAndDeform();

Scripting
Quote:
A cool thing could be if yage was detecting changes to the custom D "script" and compiling into a loadable library while the engine is running and then reloading the library. For a command prompt it may not be the best solution though.

I think it could even be usable from a command prompt. Implicitly wrap the code you've entered in a function and call it. As cool as it is though, I think implementing immediate-mode D is too ambitious. Maybe if there was already an existing project where it worked.
Back to top
View user's profile Send private message
jdrewsen



Joined: 05 Mar 2008
Posts: 21
Location: Copenhagen, Denmark

PostPosted: Tue Dec 28, 2010 6:58 am    Post subject: Reply with quote

JoeCoder wrote:

Components
I was thinking of something like this. The CollisionNode and the child ModelNode would be assigned the same RagDollModel geometry:
Code:
class CollisionNode : Node
{   Geometry shape;
   void collideAndDeform(); // simplified
}

auto collisionNode = new CollisionNode(scene);
auto modelNode = new ModelNode(collisionNode);

auto human = new Model("Human.dae");
modelNode.model = human;
collisionNode.shape = human;

human.dance(); // animates model, updates collision hull
collisionNode.collideAndDeform();



Okey. so the collisionNode would do both the collision part and the collision response handling. Most games need CollisionNodes that have simple collision handling e.g. trigger nodes. They also need CollisionNodes that have more elaborate collision handling like putting forces on a rigidbody. Doesn't this mean that since CollisionNode is in charge of collision response as well it should always contain a rigidbody just in case we would like the object to behave like a rigidbody. This in turn would mean that simple triggers also contains a rigidbody even though not used.

Of course if it is acceptable that all collidable objects also contains a (possible null) rigidbody then this design is okey. Is that the case here?


JoeCoder wrote:

Scripting
Quote:
A cool thing could be if yage was detecting changes to the custom D "script" and compiling into a loadable library while the engine is running and then reloading the library. For a command prompt it may not be the best solution though.

I think it could even be usable from a command prompt. Implicitly wrap the code you've entered in a function and call it. As cool as it is though, I think implementing immediate-mode D is too ambitious. Maybe if there was already an existing project where it worked.


Pretty ambitious yes. And I guess not of highest priority right now. I just wanted to get a feeling of the chance of adopting it if it got implemented Smile
Back to top
View user's profile Send private message
JoeCoder



Joined: 29 Oct 2005
Posts: 294

PostPosted: Tue Dec 28, 2010 10:38 am    Post subject: Reply with quote

I just realized that I've been misreading your "rigidbody" as "ragdoll" this entire time! And I'm sure I've confused you by doing so.

Quote:
Okay. so the collisionNode would do both the collision part and the collision response handling. Most games need CollisionNodes that have simple collision handling e.g. trigger nodes. They also need CollisionNodes that have more elaborate collision handling like putting forces on a rigidbody. Doesn't this mean that since CollisionNode is in charge of collision response as well it should always contain a rigidbody just in case we would like the object to behave like a rigidbody. This in turn would mean that simple triggers also contains a rigidbody even though not used.

Of course if it is acceptable that all collidable objects also contains a (possible null) rigidbody then this design is okay. Is that the case here?


I admit to having almost no experience with physics engines, so this is really helping. I think allowing for a null rigid body is a good way to go. I suppose CollisionNode would have a function to query if something is inside its geometry, and also a flag to set whether this same geometry acts as a rigid body in the physics engine. Do I understand correctly and do you see any problems?
Back to top
View user's profile Send private message
jdrewsen



Joined: 05 Mar 2008
Posts: 21
Location: Copenhagen, Denmark

PostPosted: Tue Dec 28, 2010 2:09 pm    Post subject: Reply with quote

JoeCoder wrote:
I just realized that I've been misreading your "rigidbody" as "ragdoll" this entire time! And I'm sure I've confused you by doing so.


Ok.. makes sense Smile

JoeCoder wrote:

Quote:
Okay. so the collisionNode would do both the collision part and the collision response handling. Most games need CollisionNodes that have simple collision handling e.g. trigger nodes. They also need CollisionNodes that have more elaborate collision handling like putting forces on a rigidbody. Doesn't this mean that since CollisionNode is in charge of collision response as well it should always contain a rigidbody just in case we would like the object to behave like a rigidbody. This in turn would mean that simple triggers also contains a rigidbody even though not used.

Of course if it is acceptable that all collidable objects also contains a (possible null) rigidbody then this design is okay. Is that the case here?


I admit to having almost no experience with physics engines, so this is really helping. I think allowing for a null rigid body is a good way to go. I suppose CollisionNode would have a function to query if something is inside its geometry, and also a flag to set whether this same geometry acts as a rigid body in the physics engine. Do I understand correctly and do you see any problems?


I mostly know about how ODE does it... but by quickly looking at Bullet it seems they are doing it the same way.

Collisions are handled by registering collision hulls (sphere, plane, mesh...) in Bullet. Optionally you can tell this to be simulated as a rigidbody if you want to. The you call bullet in each tick and it will handle the rigidbody simulation for you including collision/collision responses. If you want to know about collisions yourself you have to register a callback that receives the pair of colliding objects. Such an object can contain a custom pointer which could be set to an yage node.

Positions,velocities etc. are also handled in the bullet engine and of course updated upon collision responses. This means that the position, velocities in yage would have to skip its own pos, vel and use the bullet ones instead. So maybe a CollisionNode or "BulletNode" shouldn't derive from Node because it already has pos,vel,orient etc.?

Well I guess it would be best to make that decision when bullet is going to be integrated at some point Smile
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Yage All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group