GSoC - Package management with Fuel
Hi
This week has ended Google Summer of Code program 2012. I have to
thank Marcus and Carlos Ferro, my mentors, as well as Mariano,
Guille Polito, Camillo Bruni, and others, who collaborated in
different ways during these three months. Also thank, of course,
to Carla and Janko... and Google ;)
The main purpose of this mail is to summarize the work done for my
project, named "Package management with Fuel" 1.
Tools for managing code in Pharo export classes, traits, and methods
as series of source definitions, which then import by evaluating them
with the compiler. This is basically how FileOut/FileIn or Monticello
works. This is slow on large pieces of code, so if you want to install
a big package, you need a lot of time compiling it. Why should you
compile it again, given that it was all compiled when you exported the
package? The bytecodes will be the same again, you know... If you just
could load the Classes and Compiled Methods instead of compiling it
all again, you can save time.
With this goal in mind, I started working in FuelPackageLoader4 (now
Tanker3), a prototype we wrote last year, which already exports and
imports packages without using any compiler. But it has two important
issues: it wasn't capable of export source code, and it wasn't able to
tolerate class-shape changes in hierarchies (ie, a change on inst vars
in a external superclass). So, we decided to start from scratch with a
new prototype, code name Carbono. The idea was to reify how FileOut
and FileIn are done, but storing binary information so that
compilation can be avoided.
In a nutshell, Carbono stores a package into two files:
- A sources file, imitating the traditional FileOut chunk format.
- A binary file, where uses Fuel5 to serialize a series of code
definitions. Each of them knows the position in the sources file where
its source code counterpart is. For example, a class definition knows
the position of its comment (in the case it exists), and a method
definition knows the position of its source code.
A basic example, to export a couple of classes:
TAExport new
binariesOn: aBinaryStream
sourcesOn: aTextStream;
behaviors: {ClassA. ClassB};
methods: {};
initializeClasses: true;
run.
which then you can import with:
TAImport new
binariesFrom: aBinaryStream
sourcesFrom: aTextStream;
run.
Anyway, you can also choose to:
- export only binaries or only sources
- import only binaries (source code can be decompiled)
- import only sources (via #fileIn)
Mariano Martinez Peck worked a lot on it, fixing bugs, and especially
optimizing it A LOT, so I hope you will enjoy his demonstration this
Monday on ESUG Innovation Awards. Seems that he could export Seaside,
Pier and Magritte, and import them in another image, with source code,
and just in a few seconds ;) Also, some ultra confidential WikiLeaks
revealed that he could export almost all Pharo packages, and import
them in a kernel image, all with source codes but without using the
compiler :)
Note1: Tolerating class-shape changes in hierarchies is not yet done,
but it should in some days. So far, the only strategy when a change is
detected will be to recompile methods, but it may be more
sophisticated in a not-so-far future.
Note2: Although finally it was not strictly necessary, Guillermo
Polito and me wrote tests for a new ClassBuilder. It is part of Slot,
a kind of reification of objects layout made by Toon Verwaest and
Camillo Bruni. It is too long to explain it here, but the thing is
that was not an easy job, and we hope it will be integrated in Pharo.
This is a work in progress, but you can install and try it on latest
Pharo 2.0, evaluating:
Gofer it
squeaksource3: 'PharoTaskForces';
package: 'Slot';
package: 'SlotTests';
load.
(Smalltalk at: #SlotVirus) perform: #spread.
Gofer it
squeaksource3: 'PharoTaskForces';
package: 'SlotsClassBuilderIntegration';
load.
Gofer new
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) loadBleedingEdge.
Gofer new
squeaksource: 'FuelExperiments';
package: 'Carbono';
package: 'CarbonoTests';
load.
Cheers,
Martín
Great work!!! Congrats.
Saludos!
NicoPaez http://nicopaez.wordpress.com/
On Fri, Aug 24, 2012 at 5:01 AM, Martin Dias tinchodias@gmail.com wrote:
Hi
This week has ended Google Summer of Code program 2012. I have to
thank Marcus and Carlos Ferro, my mentors, as well as Mariano,
Guille Polito, Camillo Bruni, and others, who collaborated in
different ways during these three months. Also thank, of course,
to Carla and Janko... and Google ;)
The main purpose of this mail is to summarize the work done for my
project, named "Package management with Fuel" 1.
Tools for managing code in Pharo export classes, traits, and methods
as series of source definitions, which then import by evaluating them
with the compiler. This is basically how FileOut/FileIn or Monticello
works. This is slow on large pieces of code, so if you want to install
a big package, you need a lot of time compiling it. Why should you
compile it again, given that it was all compiled when you exported the
package? The bytecodes will be the same again, you know... If you just
could load the Classes and Compiled Methods instead of compiling it
all again, you can save time.
With this goal in mind, I started working in FuelPackageLoader4 (now
Tanker3), a prototype we wrote last year, which already exports and
imports packages without using any compiler. But it has two important
issues: it wasn't capable of export source code, and it wasn't able to
tolerate class-shape changes in hierarchies (ie, a change on inst vars
in a external superclass). So, we decided to start from scratch with a
new prototype, code name Carbono. The idea was to reify how FileOut
and FileIn are done, but storing binary information so that
compilation can be avoided.
In a nutshell, Carbono stores a package into two files:
- A sources file, imitating the traditional FileOut chunk format.
- A binary file, where uses Fuel5 to serialize a series of code
definitions. Each of them knows the position in the sources file where
its source code counterpart is. For example, a class definition knows
the position of its comment (in the case it exists), and a method
definition knows the position of its source code.
A basic example, to export a couple of classes:
TAExport new
binariesOn: aBinaryStream
sourcesOn: aTextStream;
behaviors: {ClassA. ClassB};
methods: {};
initializeClasses: true;
run.
which then you can import with:
TAImport new
binariesFrom: aBinaryStream
sourcesFrom: aTextStream;
run.
Anyway, you can also choose to:
- export only binaries or only sources
- import only binaries (source code can be decompiled)
- import only sources (via #fileIn)
Mariano Martinez Peck worked a lot on it, fixing bugs, and especially
optimizing it A LOT, so I hope you will enjoy his demonstration this
Monday on ESUG Innovation Awards. Seems that he could export Seaside,
Pier and Magritte, and import them in another image, with source code,
and just in a few seconds ;) Also, some ultra confidential WikiLeaks
revealed that he could export almost all Pharo packages, and import
them in a kernel image, all with source codes but without using the
compiler :)
Note1: Tolerating class-shape changes in hierarchies is not yet done,
but it should in some days. So far, the only strategy when a change is
detected will be to recompile methods, but it may be more
sophisticated in a not-so-far future.
Note2: Although finally it was not strictly necessary, Guillermo
Polito and me wrote tests for a new ClassBuilder. It is part of Slot,
a kind of reification of objects layout made by Toon Verwaest and
Camillo Bruni. It is too long to explain it here, but the thing is
that was not an easy job, and we hope it will be integrated in Pharo.
This is a work in progress, but you can install and try it on latest
Pharo 2.0, evaluating:
Gofer it
squeaksource3: 'PharoTaskForces';
package: 'Slot';
package: 'SlotTests';
load.
(Smalltalk at: #SlotVirus) perform: #spread.
Gofer it
squeaksource3: 'PharoTaskForces';
package: 'SlotsClassBuilderIntegration';
load.
Gofer new
smalltalkhubUser: 'marianopeck' project: 'Tanker';
package: 'ConfigurationOfTanker';
load.
(Smalltalk at: #ConfigurationOfTanker) loadBleedingEdge.
Gofer new
squeaksource: 'FuelExperiments';
package: 'Carbono';
package: 'CarbonoTests';
load.
Cheers,
Martín
Esug-list mailing list
Esug-list@lists.esug.org
http://lists.esug.org/mailman/listinfo/esug-list_lists.esug.org