FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

sample does not build
Goto page Previous  1, 2, 3  Next
 
Post new topic   Reply to topic     Forum Index -> Orange
View previous topic :: View next topic  
Author Message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Fri Jul 23, 2010 7:08 am    Post subject: Reply with quote

doob wrote:

I'll have a look at this. Please use the ticket system in the future: http://dsource.org/projects/orange/newticket

Will do.
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Sat Jul 24, 2010 11:02 am    Post subject: Reply with quote

jaysistar wrote:
I have fixed this and several other problems in my local copy. Next, I'll try modifying XMLArchive.d to work with Phobos, and I also might add some sort of binary archive as well. As for right now, the only new file that I have to contribute is a Makefile which handles building, cleaning, installing, and uninstalling (Mac OS X/Linux/Windows with MSYS/MinGW) How can I get the mods to you?


I copied the whole std.xml module and made the necessary modification to make it fit in Orange. Serialization works now but deserialization still doesn't work. I'll commit and push those changes so you can have a look at them if you want. About your modifications, you can make a diff using "hg diff > orange.patch" and create a new ticket and then attach the patch to the ticket.
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Sat Jul 24, 2010 3:52 pm    Post subject: Reply with quote

doob wrote:
I copied the whole std.xml module and made the necessary modification to make it fit in Orange. Serialization works now but deserialization still doesn't work. I'll commit and push those changes so you can have a look at them if you want.

Ok. I haven't seen them in the repository yet, so I guess you're still working on that.
doob wrote:
About your modifications, you can make a diff using "hg diff > orange.patch" and create a new ticket and then attach the patch to the ticket.

Ok. I'm waiting to send any diffs to you until I have your changes plus the test app (on the front page of the project) compiling.
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Sun Jul 25, 2010 3:59 am    Post subject: Reply with quote

I had some conflicts and didn't notice that my changes weren't pushed, they should be available to you now.
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Sun Jul 25, 2010 5:27 am    Post subject: Reply with quote

Deserialization works now with D2.
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Thu Jul 29, 2010 4:29 am    Post subject: Reply with quote

I added a ticket with a Makefile. The rest of your changes to make D2 and Phobos work were actually a much better fit than mine, so it wasn't really necessary to send them. I will send you some new archive formats when I'm done with them, though.

By the way, in making these new formats, I'm noticing something that puzzled me a bit. D (D2 at least) allows compile time reflection of default values if I remember correctly. Since defaults are available, why is it that XMLArchive.d throws an exception when it doesn't find a variable in the serialization? I can understand why some projects might want to do that, but I think that most projects would want the ability to add or remove variables to the object being serialized in newer versions, and still use serializations of a previous version.

That sort of flexibility is really nice to have, and I know that because I have written a C serialization library that has that ability. The main disadvantages of my library is that used a third party library (read "not known to all C programmers") to do arrays and strings (for bounds information and automatic reallocation), and it also requires you to define your structures in a separate file that isn't C (it's JSON), and then run a code generator on it. So, you can see what advantages Orange gives me: Define structures and classes in the same language in which you're coding (D) requiring no extra code generator, and no non-standard library required for array and string access. Those are huge benefits!

So, could this feature (the ability for the archive to tell the serializer to use the default value for a field that it doesn't find in the serialization) be added to Orange? (I'm adding a ticket with the bulk of this message.)
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Fri Jul 30, 2010 4:25 am    Post subject: Reply with quote

jaysistar wrote:
I added a ticket with a Makefile. The rest of your changes to make D2 and Phobos work were actually a much better fit than mine, so it wasn't really necessary to send them. I will send you some new archive formats when I'm done with them, though.

I guess I could add the makefile anyway, I guess it doesn't hurt just being there.

Quote:
By the way, in making these new formats, I'm noticing something that puzzled me a bit. D (D2 at least) allows compile time reflection of default values if I remember correctly. Since defaults are available, why is it that XMLArchive.d throws an exception when it doesn't find a variable in the serialization? I can understand why some projects might want to do that, but I think that most projects would want the ability to add or remove variables to the object being serialized in newer versions, and still use serializations of a previous version.

That sort of flexibility is really nice to have, and I know that because I have written a C serialization library that has that ability. The main disadvantages of my library is that used a third party library (read "not known to all C programmers") to do arrays and strings (for bounds information and automatic reallocation), and it also requires you to define your structures in a separate file that isn't C (it's JSON), and then run a code generator on it. So, you can see what advantages Orange gives me: Define structures and classes in the same language in which you're coding (D) requiring no extra code generator, and no non-standard library required for array and string access. Those are huge benefits!

So, could this feature (the ability for the archive to tell the serializer to use the default value for a field that it doesn't find in the serialization) be added to Orange? (I'm adding a ticket with the bulk of this message.)

I've replied in ticket and I'm replying here as well:

Fist, the plan was to be able to pass in a bool indicating if the serializer/archive should throw or not when not finding a value. I guess I never implemented that. Second, I'm planning to add some sort of version handling for classes and structs, I've just haven't figured out how I want that to behave like. I'm open to suggestions. About the default values, I guess there's no problem in D1 either because all variables are default initialized in D.
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Sun Aug 01, 2010 7:14 pm    Post subject: Reply with quote

Perhaps a version number for the whole archive given by the application would suffice?
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Mon Aug 02, 2010 4:38 am    Post subject: Reply with quote

I was thinking more in the line of a version per class/struct.
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Mon Aug 02, 2010 8:27 pm    Post subject: Reply with quote

I know that you said that you don't know how you want that to work yet, but I'm going to ask anyway Smile: From where do those version numbers come? Do they come from the application, or are the numbers derived in some way from the structure of the class/struct (like a hash function would provide)?

I know that I have many large arrays of light weight structs in my apps (vectors and matrices and the like), and I'm wondering if a version per class/struct is too much overhead. Are these versions optional or required?
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Tue Aug 03, 2010 2:35 am    Post subject: Reply with quote

jaysistar wrote:
I know that you said that you don't know how you want that to work yet, but I'm going to ask anyway Smile: From where do those version numbers come? Do they come from the application, or are the numbers derived in some way from the structure of the class/struct (like a hash function would provide)?

I was thinking something like this:
Code:
class Foo
{
    mixin Version!("0.0.1");
}

Quote:
I know that I have many large arrays of light weight structs in my apps (vectors and matrices and the like), and I'm wondering if a version per class/struct is too much overhead. Are these versions optional or required?

Of course they would be optional. Would that be too much overhead if the Version mixin adds one extra static variable?

A different interface could look like something like this:
Code:
serializer.registerVersion(Foo.classinfo.name, "0.0.1");
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Tue Aug 03, 2010 3:51 am    Post subject: Reply with quote

doob wrote:
Of course they would be optional. Would that be too much overhead if the Version mixin adds one extra static variable?
Ok, static variables don't bother me if they're only in the archive once. I know how an app would use a version, but I'm a bit unclear about how the serializer would use it, unless, maybe, it had all possible versions of the class/struct.

In other words, I'm certainly not opposed to optional versioning, but if each field is optionally included in the archive (if it's not there, it's default value is assigned), then I'm not sure how knowing the version would benefit the serializer. The only benefit of which I could think is that the exception throwing mechanism for non-present fields in the archive could be automatically turned on or off if the version were different or the same respectively. Was this your reasoning, or do you have a different case for versioning?
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Thu Aug 05, 2010 4:46 am    Post subject: Reply with quote

I haven't given this that much thought so I don't know. I'm going to look at other serializers and see how they handle this, hopefully getting some ideas how I want to handle different versions in Orange.
Back to top
View user's profile Send private message
jaysistar



Joined: 30 Jun 2010
Posts: 31

PostPosted: Fri Aug 06, 2010 4:55 am    Post subject: Reply with quote

From what I can tell, most serializers are poor at versioning. To my knowledge, only Google's ProtoBuf and Hadoop's Avro handle versions well.

ProtoBuf doesn't claim to be a serializer by name, but it is; It's just not an "auto"-serializer. The problem with protobuf is that it tries to offer language interoperability over auto-serialization, and therefore has its own file format to define data structures, which is a scheme similar to IDLs to define callable procedures for RPC. You must have a language-specific code generator to use it, and the code that it generates doesn't always fit easily into your app. In the case of C++, which has no property variables, you have to use explicit inlined getters and setters for every value of the class that it generates, and you can't inherit that class, so you can see why auto-serializers would be more appealing.

Regardless of these oddities, ProtoBuf handles versioning very well. In its definition file, it requires that a "key" number be assigned to each variable, which is specified in the stream adjacent to the serialized value of that data. In the definition file, you may also specify variables as "optional" to facilitate version changing and default for optional values. There is no explicit version number, however.

Perhaps a better versioning system (still with no explicit version number, though) is that of Hadoop's Avro, which separates the data from the metadata a bit more. The metadata is JSON at the beginning of the serialized stream. There's no coded mapping between variable names and keys, like ProtoBuf, just the variable names, and their types. You may have noticed that this format gives you more information than ProtoBuf (types are specifed and variable names are given instead of keys). You can decipher an Avro file by itself, but need the ProtoBuf definition file to make complete sense of a ProtoBuf encoding.

Avro does versioning by comparing the metadata definition that it has to the JSON in the stream being deserailized. When requesting data from the Avro backend, the user will specify a "projection" (think SQL: SELECT <projection> FROM <table> etc.) The projection is JSON (or a parsed JSON tree structure) describing what the user is expecting. The projection may specify variables that don't exist in the incoming stream, and when that happens, the default value is used (Defaults can also be specifed in the projection JSON to override a type's default).

While neither of these have versions, I can see that versions could provide a way to tell (automatically) if Orange Archive classes should throw exceptions or return normally (perhaps even a callback) when they don't find a variable in an archive. I do think, however, that a more manual approach (like a parameter to the deserialize() function) might feel a bit less invasive. The main feature that I like about Orange is the fact that I can take my data as it is, and pass it to the serialize function, and it works. Yes, I do have to have an opEqual defined, but that's part of the language, so it doesn't intrude. The only essential feature (for me) that it's missing right now is this "don't throw" mechanism. If it had that, it would be a perfect fit (for me).
Back to top
View user's profile Send private message
doob



Joined: 06 Jan 2007
Posts: 367

PostPosted: Sat Aug 07, 2010 7:41 am    Post subject: Reply with quote

I like that idea of optional variables. I didn't understand much of the metadata in JSON, it sounds complicated. About the opEquals, that is just to overwrite the default comparison to show that two objects are considered equal, i.e. that they contain the same data. opEquals is not necessary for making the serializer work. As default opEquals compare the addresses of two objects and they will not be the same when the serialized object is deserialized.

About the optional throwing, could you put that as a enhancement request in the ticket system so I don't forget about it.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic     Forum Index -> Orange All times are GMT - 6 Hours
Goto page Previous  1, 2, 3  Next
Page 2 of 3

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group