ESUG SummerTalk - Fuel, binary object serializer
Hi folks. I am really happy to announce that ESUG is sponsoring me for Fuel
development through the ESUG SummerTalk. I am Martin Dias, a student at
Buenos Aires, Argentina. The idea behind this SummerTalk is to implement
Fuel, a binary, fast and general-purpose object graph serializer in
Pharohttp://www.pharo-project.org/.
It is based on VisualWorks' Parcels ideas.
Actually, the project has already started since several months. Tristan
Bourgois and I started with the project while doing an internship with RMoD,
INRIA http://rmod.lille.inria.fr/web/pier. Since a couple of months, Mariano
Martinez Peck http://marianopeck.wordpress.com/ joined the team, and now
he is the official mentor in the SummerTalk.
ESUG website for SummertTalk:
http://www.esug.org/wiki/pier/Promotion/SummerTalk/SummerTalk2011
The website with all the necessary information is here:
http://rmod.lille.inria.fr/web/pier/software/Fuel
It even includes slides explaining the algorithm. In addition, a paper is in
progress.
For the moment, Fuel already provides the following features:
- Fast pickle format. It is much faster to materialize than to serialize.
- Correctly support class reshape (when the class of serialized objects has
changed). - Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize. - Be able to completely serialize classes and traits (not just a global
name). - Support cycles and avoid duplicates in the graph.
- Integration to Moose http://www.moosetechnology.org/ with an extension
to export and import their models. - Detection of globals: for example if you serialize Transcript, it is not
duplicated and instead managed as a global reference. - Solve common problems like Set rehash.
- Buffered writing: we use a buffered write stream for the serialization
part (thanks Sven!). - No need of special support from the VM.
- Try to have a good object oriented design.
- Well tested (about 120 tests, for the moment).
- Large set of benchmarks (even benchmarks for Moose extension).
And of course, there are a lot features for the future. You can see some of
them in the website and some in the issue tracker:
http://code.google.com/p/fuel/issues/list
We really appreciate all kind of feedback and comments. If you want to try
it, check in the website how to do it. It is extremely easy.
Once again, I want to thank a lot to ESUG for sponsoring the project. I plan
to create a "news" section in the website with some RSS. I will keep you
informed.
Best regards,
Martin
2011/5/24 Martin Dias tinchodias@gmail.com:
Hi folks. I am really happy to announce that ESUG is sponsoring me for Fuel
development through the ESUG SummerTalk. I am Martin Dias, a student at
Buenos Aires, Argentina. The idea behind this SummerTalk is to implement
Fuel, a binary, fast and general-purpose object graph serializer in Pharo.
It is based on VisualWorks' Parcels ideas.
Please excuse me if this is the wrong forum to discuss things.
Actually, the project has already started since several months. Tristan
Bourgois and I started with the project while doing an internship with RMoD,
INRIA. Since a couple of months, Mariano Martinez Peck joined the team, and
now he is the official mentor in the SummerTalk.
ESUG website for SummertTalk:
http://www.esug.org/wiki/pier/Promotion/SummerTalk/SummerTalk2011
The website with all the necessary information is here:
http://rmod.lille.inria.fr/web/pier/software/Fuel
It even includes slides explaining the algorithm. In addition, a paper is in
progress.
Could you make the slides available in some other format than Flash?
For the moment, Fuel already provides the following features:
- Fast pickle format. It is much faster to materialize than to serialize.
What has led to the conclusion that materialization is more important
than serialization? I can image scenarios when there is a one to one
relationship and scenarios where serialization is more important (e.g.
session replication).
- Correctly support class reshape (when the class of serialized objects has
changed).
So what do you do when an instance variable was added? Set it to nil
and hope that everything will continue to work?
- Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize.
Really? You serialize Socket, Process, FileStream and something
meaningful happens?
- Be able to completely serialize classes and traits (not just a global
name). - Support cycles and avoid duplicates in the graph.
- Integration to Moose with an extension to export and import their models.
- Detection of globals: for example if you serialize Transcript, it is not
duplicated and instead managed as a global reference. - Solve common problems like Set rehash.
- Buffered writing: we use a buffered write stream for the serialization
part (thanks Sven!). - No need of special support from the VM.
- Try to have a good object oriented design.
- Well tested (about 120 tests, for the moment).
- Large set of benchmarks (even benchmarks for Moose extension).
And of course, there are a lot features for the future. You can see some of
them in the website and some in the issue tracker:
http://code.google.com/p/fuel/issues/list
We really appreciate all kind of feedback and comments. If you want to try
it, check in the website how to do it. It is extremely easy.
Once again, I want to thank a lot to ESUG for sponsoring the project. I plan
to create a "news" section in the website with some RSS. I will keep you
informed.
Cheers
Philippe
On Tue, May 24, 2011 at 11:39 PM, Philippe Marschall <
philippe.marschall@gmail.com> wrote:
Could you make the slides available in some other format than Flash?
I second that.
...
- Fast pickle format. It is much faster to materialize than to serialize.
What has led to the conclusion that materialization is more important
than serialization? I
I don't think Martin made a claim about importance, just current
implementation behavior. One horse will be faster than another. Currently,
the materialization horse is faster than the serialization horse.
- Correctly support class reshape (when the class of serialized objects
has
changed).
So what do you do when an instance variable was added? Set it to nil
and hope that everything will continue to work?
Of course. One might hope that the developer that added an instance
variable to a system which had existing instanciated objects would provide
suitable lazy initialization as needed. There is no magic.
- Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize.
Really? You serialize Socket, Process, FileStream and something
meaningful happens?
Again, no magic. One would hope an interface object that became detached
from it's service would know how to reattach.
ttfn, Steve
Steve Cline cline@acm.org http://www.clines.org
http://www.linkedin.com/in/stevecline
"Do what's right, and try to get along with people, in that order"
- Ezra Taft Benson
Hi Philippe. I don't know why but your emails always go to spam in my
account :(
Actually, the project has already started since several months. Tristan
Bourgois and I started with the project while doing an internship with
RMoD,
INRIA. Since a couple of months, Mariano Martinez Peck joined the team,
and
now he is the official mentor in the SummerTalk.
ESUG website for SummertTalk:
http://www.esug.org/wiki/pier/Promotion/SummerTalk/SummerTalk2011
The website with all the necessary information is here:
http://rmod.lille.inria.fr/web/pier/software/Fuel
It even includes slides explaining the algorithm. In addition, a paper is
in
progress.
Could you make the slides available in some other format than Flash?
Ok, Martin will do it and upload them directly to the website.
For the moment, Fuel already provides the following features:
- Fast pickle format. It is much faster to materialize than to serialize.
What has led to the conclusion that materialization is more important
than serialization?
One of the most important uses we want to do with Fuel (in a future) is to
be able to use it for Monticello (to replace mzc).
The idea in addition is to be able to boostrap a really small pharo image
(hetzel) and be able to load stuff without needing a compiler.
I all those cases we assume that you serialize only once or few times and
you materialize much more times.
But of course, that does not apply to all cases, like you said about session
replication.
I can image scenarios when there is a one to one
relationship and scenarios where serialization is more important (e.g.
session replication).
- Correctly support class reshape (when the class of serialized objects
has
changed).
So what do you do when an instance variable was added? Set it to nil
and hope that everything will continue to work?
In that example yes. But what we meant is that right now the inst var names
are bening encoded also. So at materializatio ntime we will be able to deal
with them.
Not all scnearios are developed right now, but the base is there. At some
point, the only solution has to be done by the user. For example, you should
implement
#updateFrom: aVersion to: anotherVersion: anotherVersion object: anObject
or something like that....
So...we meant that we store the instVar names and we support some type of
changes.
In fact, we have reified FLInstanceVariablesMapping. So maybe we can do
something with that in a future.
- Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize.
Really? You serialize Socket, Process, FileStream and something
meaningful happens?
Of course there are classes whose instances doesn't make sense once to load
them back in the image. Our comment was literally: we can (ok, we should be
able to) serialize them and materialize them without problem. Now....whether
those instances are correct or still valid and meaningful in the current
image is another problem ;)
One of the future features will be to be able to implement
#postMaterializationAction or something like that could be executed after
materialization.
Sockets could try to get a new socket from the OS, Processes could be
restarted and rescheduled? I have no idea. If you have, please let us know.
Thanks for the execellent questions :)
--
Mariano
http://marianopeck.wordpress.com
- Correctly support class reshape (when the class of serialized objects
has
changed).
So what do you do when an instance variable was added? Set it to nil
and hope that everything will continue to work?
Of course. One might hope that the developer that added an instance
variable to a system which had existing instanciated objects would provide
suitable lazy initialization as needed. There is no magic.
Exactly. Read the answer I have just send.
- Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize.
Really? You serialize Socket, Process, FileStream and something
meaningful happens?
Again, no magic. One would hope an interface object that became detached
from it's service would know how to reattach.
Exactly. Ideally, a class will be able to implement
#prepareToBeSerializer and #postMaterializationAction
or stuff like that. So, what we want to provide is the infrastructure, the
hooks.
Then, we shoud identify those base classes that MUST have something like
that like the cases pointed out by Philippe
For example, the same happes with Set and Dictionaries. They have to be
rehashes once they are materialized.
ttfn, Steve
Steve Cline cline@acm.org http://www.clines.org
http://www.linkedin.com/in/stevecline
"Do what's right, and try to get along with people, in that order"
- Ezra Taft Benson
Esug-list mailing list
Esug-list@lists.esug.org
http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org
--
Mariano
http://marianopeck.wordpress.com
At Wed, 25 May 2011 15:28:00 +0200,
Mariano Martinez Peck wrote:
One of the most important uses we want to do with Fuel (in a future) is to be able to use it for Monticello (to replace mzc).
The idea in addition is to be able to boostrap a really small pharo image (hetzel) and be able to load stuff without needing a compiler.
Sounds interesting! Can I learn about "hetzel" somewhere?
-- Yoshiki
Hi
On Wed, May 25, 2011 at 10:28 AM, Mariano Martinez Peck <
marianopeck@gmail.com> wrote:
Hi Philippe. I don't know why but your emails always go to spam in my
account :(
Actually, the project has already started since several months. Tristan
Bourgois and I started with the project while doing an internship with
RMoD,
INRIA. Since a couple of months, Mariano Martinez Peck joined the team,
and
now he is the official mentor in the SummerTalk.
ESUG website for SummertTalk:
http://www.esug.org/wiki/pier/Promotion/SummerTalk/SummerTalk2011
The website with all the necessary information is here:
http://rmod.lille.inria.fr/web/pier/software/Fuel
It even includes slides explaining the algorithm. In addition, a paper
is in
progress.
Could you make the slides available in some other format than Flash?
Ok, Martin will do it and upload them directly to the website.
Yes, sorry. I kept the embedded slides but I added links to the pdf
"sources".
For the moment, Fuel already provides the following features:
- Fast pickle format. It is much faster to materialize than to
serialize.
What has led to the conclusion that materialization is more important
than serialization?
One of the most important uses we want to do with Fuel (in a future) is to
be able to use it for Monticello (to replace mzc).
The idea in addition is to be able to boostrap a really small pharo image
(hetzel) and be able to load stuff without needing a compiler.
I all those cases we assume that you serialize only once or few times and
you materialize much more times.
But of course, that does not apply to all cases, like you said about
session replication.
Actually, despite we have this hypothesis (de-serialization is done much
more often than serialization), I often think it is not something really
essential to the project. It was an early decision to base on the Parcel's
algorithm, which has this feature, but I believe we can eventually implement
an alternative strategy that favors serialization performance over
deserialization.
I think other more essential features (or goals!) are:
- any object can be serialized.
- binary class load, without compilation.
- fast and focused on just one dialect, don't worry about an inter-platform
format. - flexibility, for selecting the graph to serialize.
I can image scenarios when there is a one to one
relationship and scenarios where serialization is more important (e.g.
session replication).
- Correctly support class reshape (when the class of serialized objects
has
changed).
So what do you do when an instance variable was added? Set it to nil
and hope that everything will continue to work?
In that example yes. But what we meant is that right now the inst var names
are bening encoded also. So at materializatio ntime we will be able to deal
with them.
Not all scnearios are developed right now, but the base is there. At some
point, the only solution has to be done by the user. For example, you should
implement
#updateFrom: aVersion to: anotherVersion: anotherVersion object: anObject
or something like that....
So...we meant that we store the instVar names and we support some type of
changes.
In fact, we have reified FLInstanceVariablesMapping. So maybe we can do
something with that in a future.
I want to let the user configure an initialization block.
- Serialize ANY kind of object. For the moment there is no object to our
knowledge that we cannot serialize and materialize.
Really? You serialize Socket, Process, FileStream and something
meaningful happens?
Of course there are classes whose instances doesn't make sense once to load
them back in the image. Our comment was literally: we can (ok, we should be
able to) serialize them and materialize them without problem. Now....whether
those instances are correct or still valid and meaningful in the current
image is another problem ;)
One of the future features will be to be able to implement
#postMaterializationAction or something like that could be executed after
materialization.
Sockets could try to get a new socket from the OS, Processes could be
restarted and rescheduled? I have no idea. If you have, please let us know.
Yes... maybe I was too much optimistic with this "ANY". I think you can
serialize and deserialize instances of these classes but probably they need
something to be "meaningful".
Thanks for the execellent questions :)
Thanks!
Martin
--
Mariano
http://marianopeck.wordpress.com