How to output the AST nodes of an OpenSCAD scripting file?
Dear Jordan,
Thank you very much for your help.
What are you really trying to accomplish? I know you've talked about emitting Python, but for what purpose?
We want to convert OpenSCAD scripts into CadQuery Python script files. Previously, we initiated a discussion titled "About Automatically Translating OpenSCAD Scripting Code into CadQuery Scripting Code (Python Code)." At that time, you also provided a suggestion: "Implementing the lexical analysis and the parser, or extracting the AST from OpenSCAD somehow, plus obvious implementations of primitives, will get you a modest fraction of OpenSCAD programs. Driving that fraction up will take a lot more."
Just to be clear, what you're looking at is a CSG dump, not an AST dump. It's a dump of the generated geometry, not a dump of the program that generates it. That makes it a lot easier to process, but of course it doesn't have the same flexibility.
Yes, you are right. It is a CSG dump, not an AST dump. We have now pinpointed the code for parsing *.scad files in MainWindow.cc under MainWindow::parseTopLevelDocument() with the following statement:
this->rootFile = parse(this->parsedFile, fulltext, fname, fname, false) ? this->parsedFile : nullptr;
The function Parameters Parameters::parse(...) reads quite cryptically. I think it might be possible to extract the actual AST node information from this function, but it seems to require a deeper understanding of the code. Is it feasible to easily obtain information about AST nodes from the existing program?
We still prefer to work on AST nodes for the code translation.
BTW, we could get the node-level information of the CSG dump in the souce code. The seconde column shows the node-level information.
0 0root()
1 1 difference()
2 2 cube(size = [10, 10, 10], center = false)
3 2 sphere($fn = 0, $fa = 12, $fs = 2, r = 7)
4 1 translate multmatrix([[1, 0, 0, 15], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
5 2 union()
6 3 cube(size = [10, 10, 10], center = false)
7 3 sphere($fn = 0, $fa = 12, $fs = 2, r = 7)
8 1 translate multmatrix([[1, 0, 0, 0], [0, 1, 0, 15], [0, 0, 1, 0], [0, 0, 0, 1]])
9 2 minkowski(convexity = 0)
10 3 cube(size = [5, 5, 5], center = false)
11 3 sphere($fn = 0, $fa = 12, $fs = 2, r = 3)
12 1 translate multmatrix([[1, 0, 0, 15], [0, 1, 0, 15], [0, 0, 1, 0], [0, 0, 0, 1]])
13 2 hull()
14 3 cube(size = [5, 5, 5], center = false)
15 3 translate multmatrix([[1, 0, 0, 10], [0, 1, 0, 10], [0, 0, 1, 0], [0, 0, 0, 1]])
16 4 sphere($fn = 0, $fa = 12, $fs = 2, r = 3)
17 1 difference()
18 2 module mymodule
19 3 translate multmatrix([[1, 0, 0, 10], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
20 4 cube(size = [5, 5, 5], center = false)
21 3 rotate multmatrix([[1, 0, 0, 0], [0, 0.707107, -0.707107, 0], [0, 0.707107, 0.707107, 0], [0, 0, 0, 1]])
22 4 sphere($fn = 0, $fa = 12, $fs = 2, r = 10)
23 2cylinder($fn = 0, $fa = 12, $fs = 2, h = 20, r1 = 5, r2 = 5, center = false)
Best regards,
Youbao
-----Original Messages-----
From:"Jordan Brown" openscad@jordan.maileater.net
Sent Time:2025-03-30 08:08:47 (Sunday)
To: zhangyoubao@siom.ac.cn, "OpenSCAD general discussion Mailing-list" discuss@lists.openscad.org
Cc: "Marius Kintel" marius@kintel.net
Subject: Re: [OpenSCAD] Re: How to output the AST nodes of an OpenSCAD scripting file?
On 3/29/2025 2:15 AM, zhangyoubao@siom.ac.cn wrote:
Thank you very much for your help. Now, we could get all the AST nodes information in the function of NodeVisitor::traverse(...).
Just to be clear, what you're looking at is a CSG dump, not an AST dump. It's a dump of the generated geometry, not a dump of the program that generates it. That makes it a lot easier to process, but of course it doesn't have the same flexibility.
- Are there any built-in node-level information for the individual node? The AbstractNode's index() is just the node's index, not the node's level. Do we have to implement ourselves?
You've probably looked at the definition of AbstractNode more recently than I have :-)
I don't see any level stored. I doubt that OpenSCAD would have any use for it in its own processing. For a CSG dump I think it just recursively walks the tree, and increments the indent whenever it descends.
- For translate and rotate, could we keep the original parameters information, instead of multmatrix([[1, 0, 0, 15], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]) conversion? In our case, translate([15, 0, 0]) instead of multmatrix([[1, 0, 0, 15], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]).
You could certainly modify it to retain the original parameters (probably through creating subclasses of TransformNode, each of which had the right members), but it wouldn't be useful for OpenSCAD's core mission. Including those in a CSG dump would actually make CSG dumps harder to process, because the consumer would have to know about all of the possible transforms, not just about multmatrix.
What are you really trying to accomplish? I know you've talked about emitting Python, but for what purpose?
Yes, you are right. It is a CSG dump, not an AST dump. We have now pinpointed the code for parsing *.scad files in MainWindow.cc under MainWindow::parseTopLevelDocument() with the following statement:
this->rootFile = parse(this->parsedFile, fulltext, fname, fname, false) ? this->parsedFile : nullptr;
The function Parameters Parameters::parse(...) reads quite cryptically. I think it might be possible to extract the actual AST node information from this function, but it seems to require a deeper understanding of the code. Is it feasible to easily obtain information about AST nodes from the existing program?
One tip: Don’t use the GUI code when prototyping this; it’s a lot more complex than the cmd-line code paths:
The source code is parsed into an AST here: https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/openscad.cc#L598
..and dumped to our AST file format here:
https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/openscad.cc#L447-L452
To dump the AST into your own format, you really just have to reimplement ASTNode::print() for each of the node types.
If you want to be clean, you could even add your own file format to the FileFormat enum, and isolate all your code in one place:
https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/io/export.h#L29-L49
-Marius
Thank you very much, Marius.
I wonder how many node types are there in OpenSCAD? Are they all defined in node.h, NodeVisitor.h, and node_clone.cc?
If we have implemented the "print" function for CubeNode and SphereNode, what would the general code architecture look like for parsing an OpenSCAD file that only contains cube and sphere commands? For example:
/example.scad**/
cube([10, 10, 10]);
sphere(7);
/********************/
Could you please just give me a very simple framework (not the code) example just for CubeNode and SphereNode?
You are right, the GUI code is extremely obscure, and it is quite tedious to trace the code execution order.
Best regards,
Youbao
-----Original Messages-----
From:"Marius Kintel" marius@kintel.net
Sent Time:2025-03-31 23:39:13 (Monday)
To: "OpenSCAD general discussion Mailing-list" discuss@lists.openscad.org
Cc: "Jordan Brown" openscad@jordan.maileater.net, zhangyoubao@siom.ac.cn
Subject: Re: [OpenSCAD] How to output the AST nodes of an OpenSCAD scripting file?
One tip: Don’t use the GUI code when prototyping this; it’s a lot more complex than the cmd-line code paths:
The source code is parsed into an AST here: https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/openscad.cc#L598
..and dumped to our AST file format here:
https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/openscad.cc#L447-L452
To dump the AST into your own format, you really just have to reimplement ASTNode::print() for each of the node types.
If you want to be clean, you could even add your own file format to the FileFormat enum, and isolate all your code in one place:
https://github.com/openscad/openscad/blob/340aab891c36ea00703620a0b078b97acd4225a6/src/io/export.h#L29-L49
-Marius
On Mar 31, 2025, at 12:34, zhangyoubao@siom.ac.cn wrote:
I wonder how many node types are there in OpenSCAD? Are they all defined in node.h, NodeVisitor.h, and node_clone.cc?
Not too many, around 20 I think, mostly for functions and expression. All object like cube and sphere are just built-in modules and the AST doesn’t need separate nodes for them, they’re just names in the ModuleInstantiation node.
The subtypes are spread out over numerous files. Just search for all subtypes of ASTNode.
If we have implemented the "print" function for CubeNode and SphereNode, what would the general code architecture look like for parsing an OpenSCAD file that only contains cube and sphere commands? For example:
/example.scad**/
cube([10, 10, 10]);
sphere(7);
/********************/
Not sure I understand what you want.
In ModuleInstantiation::print(), you can extract the name of the module (sphere, cube) from modname, and the arguments from arguments. One the print() function is implemented, you can simply run openscad myfilescad -o out.ast, and observe your changes.
-Marius
On 3/31/2025 8:26 AM, zhangyoubao@siom.ac.cn wrote:
We want to convert OpenSCAD scripts into CadQuery Python script files.
Assuming that you want to retain the program's logic, rather than just
the shapes that it generates, you want to work from AST.
You might think that once you have an AST, you could just emit Python
syntax for that AST and be done. You'd be wrong. The problem is that
AST will get syntax out of the way, but will not help you at all with
semantics.
As a trivial example, consider the "+" operator as applied to vectors.
In Python, + does vector concatenation. In OpenSCAD, + does vector
addition. In Python, [1,2,3]+[4,5,6] yields [1, 2, 3, 4, 5, 6]; in
OpenSCAD it yields [5,7,9].
The semantics of variables are much trickier. For instance, an obvious
translation of this program:
x = 1;
echo(x);
if (x != 0) {
echo(x);
x = 2;
echo(x);
}
echo(x);
would produce 1, 1, 2, 2. In OpenSCAD it will produce 1, 2, 2, 1.
There are good reasons and bad for that ... interesting ... behavior,
but it will take a great deal of care to get it right, and if you don't
get it right then some small set of OpenSCAD programs will misbehave.
There are a lot of subtle behaviors that go into the operation of
almost any language, and OpenSCAD is no exception.
The function Parameters Parameters::parse(...) reads quite
cryptically. I think it might be possible to extract the actual AST
node information from this function, but it seems to require a deeper
understanding of the code. Is it feasible to easily obtain information
about AST nodes from the existing program?
After the call to parse() that Marius pointed you at, root_file.scope is
the top of the AST. In particular, root_file.scope.assignments is a
list of assignments, root_file.scope.moduleInstantiations is a list of
module instantiations, root_file.scope.functions is a list of functions
defined, and root_file.scope.modules is a list of modules defined.
You'd walk the tree down from that.
Dear Jordan and Marius,
Thank you for your help on the ASTNode things.
By inserting the following post-traversal processing code into the function “Response NodeVisitor::traverse(const AbstractNode& node, const State& state)”, we can now automatically generate the corresponding CadQuery Python code from a CSG-perspective:
CadQueryASTGenerator& generator = CadQueryASTGenerator::getInstance();
if (generator.isASTGenerationEnabled()) {
generator.processNode(node, node.index());
}
We've observed that the NodeVisitor class implements visit functions for 20 subclasses that inherit from the AbstractNode class (shown in the green section of the diagram below).
Next, we want to automatically generate the corresponding CadQuery Python code from an AST-perspective. Our general idea is as follows, and we'd like Jordan and Marius to give some suggestion and help.
- Referring to the implementation and inheritance relationships of the "class AbstractNode":class AbstractNode : public BaseVisitable, public std::enable_shared_from_this<AbstractNode>
We'll modify the "class ASTNode" :
class ASTNode : public BaseVisitable, public std::enable_shared_from_this<AbstractNode>
And add the following functions to the ASTNode class:
VISITABLE();
name();
verbose_name();
toString();
getChildren();
-
Implement related functions and operations in the ASTNode's subclasses similar to those in the AbstractNode subclasses. For example, in the ModuleInstantiation subclass of ASTNode, implement:
VISITABLE();
name();
verbose_name();
toString(); -
Following the pattern of the "class NodeVisitor", create an "class ASTNodeVisitor" and implement the function:
Response NodeVisitor::traverse(const ASTNode& node, const State& state); -
Create subclasses of ASTNodeVisitor, override the corresponding visit functions, and implement the desired functionality.
The classes inheriting from AbstractNode and ASTNode are shown in the diagram below.
Thank you very much for your help.
Best regards,
Youbao
On 4/14/2025 8:51 AM, zhangyoubao@siom.ac.cn wrote:
We'll modify the "class ASTNode" :
class ASTNode : public BaseVisitable, public
std::enable_shared_from_this<AbstractNode>
And add the following functions to the ASTNode class:
VISITABLE();
name();
verbose_name();
toString();
getChildren();
I should note that I'm a C++ beginner and my OO experience overall is
somewhat limited.
The visitor pattern would probably work. I'm not a big fan of it; it
violates my notions of information hiding and modularity.
I would instead follow the pattern used by ASTNode.print(), where each
subclass implements an appropriate print() method that knows how to
print that particular subclass.
Note that ASTNodes often do not have direct ASTNode children.
SourceFile, for instance - which is the root of the tree - contains a
LocalScope, and LocalScope contains an AssignmentList containing all of
the assignments, a vector of ModuleInstantiations that contains the
module instantiations, and vectors of UserFunction and UserModule that
contain those definitions. Similarly, ModuleInstantiation contains an
AssignmentList of the module arguments, and a LocalScope describing the
body. I don't know how well that heterogeneous data structure fits into
the visitor pattern. (Again, noting that I have limited OO
experience... when I say "I don't know" there, it means that I don't
know; it's not a euphemism for "it doesn't work". It's just not
instantly obvious to me that it does work.)
Dear Jordan,
Thank you for your email.
ASTNode.print() is indeed very useful for exploring information about ASTNodes, but we feel that generating PythonASTNode and further generating Python code with the ASTNodeVisitor::traverse function is more appropriate.
On OpenSCAD's pull request list, there are two pull requests currently:
- AST & visitor green refactoring #1641 .
- AST visitation #1743 .
Marius has been working on these pull requests for quite a long time and has completed a lot of the infrastructure work. If Marius could provide some explanations about the current progress and the future plans, it would be very helpful. I would be very happy to give a hand if there’s something I am capable of doing.
Best regards,
Youbao
-----Original Messages-----
From:"Jordan Brown via Discuss" discuss@lists.openscad.org
Sent Time:2025-04-15 00:28:55 (Tuesday)
To: zhangyoubao@siom.ac.cn, "OpenSCAD general discussion Mailing-list" discuss@lists.openscad.org
Cc: "Jordan Brown" openscad@jordan.maileater.net
Subject: [OpenSCAD] Re: How to output the AST nodes of an OpenSCAD scripting file?
On 4/14/2025 8:51 AM, zhangyoubao@siom.ac.cn wrote:
We'll modify the "class ASTNode" :
class ASTNode : public BaseVisitable, public std::enable_shared_from_this<AbstractNode>
And add the following functions to the ASTNode class:
VISITABLE();
name();
verbose_name();
toString();
getChildren();
I should note that I'm a C++ beginner and my OO experience overall is somewhat limited.
The visitor pattern would probably work. I'm not a big fan of it; it violates my notions of information hiding and modularity.
I would instead follow the pattern used by ASTNode.print(), where each subclass implements an appropriate print() method that knows how to print that particular subclass.
Note that ASTNodes often do not have direct ASTNode children. SourceFile, for instance - which is the root of the tree - contains a LocalScope, and LocalScope contains an AssignmentList containing all of the assignments, a vector of ModuleInstantiations that contains the module instantiations, and vectors of UserFunction and UserModule that contain those definitions. Similarly, ModuleInstantiation contains an AssignmentList of the module arguments, and a LocalScope describing the body. I don't know how well that heterogeneous data structure fits into the visitor pattern. (Again, noting that I have limited OO experience... when I say "I don't know" there, it means that I don't know; it's not a euphemism for "it doesn't work". It's just not instantly obvious to me that it does work.)
As you pointed out, I started refactoring the AST node hierarchy to being visitable and using that to implement various types of processing in https://github.com/openscad/openscad/pull/1743
I haven’t had the time to revisit this in a number of years, as doing this has limited impact on OpenSCAD itself.
I think the approach makes sense, but this is likely significant work to design, implement and test.
A good measure of success would be the ability to dump the AST in .scad format using the new visitor pattern. I think the main challenge in doing this is to be able to preserve include and use statements.
I have very limited time to look into these details, but I think a good approach would be to use the existing test framework to validate that any visitation implementation works as expected.
Also note that the current code was written in C++11, and there may be more elegant ways of solving some of this using C++17.
Kind Regards,
-Marius
On Apr 14, 2025, at 11:51, zhangyoubao@siom.ac.cn wrote:
Dear Jordan and Marius,
Thank you for your help on the ASTNode things.
By inserting the following post-traversal processing code into the function “Response NodeVisitor::traverse(const AbstractNode& node, const State& state)”, we can now automatically generate the corresponding CadQuery Python code from a CSG-perspective:
CadQueryASTGenerator& generator = CadQueryASTGenerator::getInstance();
if (generator.isASTGenerationEnabled()) {
generator.processNode(node, node.index());
}
We've observed that the NodeVisitor class implements visit functions for 20 subclasses that inherit from the AbstractNode class (shown in the green section of the diagram below).
Next, we want to automatically generate the corresponding CadQuery Python code from an AST-perspective. Our general idea is as follows, and we'd like Jordan and Marius to give some suggestion and help.
- Referring to the implementation and inheritance relationships of the "class AbstractNode":class AbstractNode : public BaseVisitable, public std::enable_shared_from_this<AbstractNode>
We'll modify the "class ASTNode" :
class ASTNode : public BaseVisitable, public std::enable_shared_from_this<AbstractNode>
And add the following functions to the ASTNode class:
VISITABLE();
name();
verbose_name();
toString();
getChildren();
-
Implement related functions and operations in the ASTNode's subclasses similar to those in the AbstractNode subclasses. For example, in the ModuleInstantiation subclass of ASTNode, implement:
VISITABLE();
name();
verbose_name();
toString(); -
Following the pattern of the "class NodeVisitor", create an "class ASTNodeVisitor" and implement the function:
Response NodeVisitor::traverse(const ASTNode& node, const State& state); -
Create subclasses of ASTNodeVisitor, override the corresponding visit functions, and implement the desired functionality.
The classes inheriting from AbstractNode and ASTNode are shown in the diagram below.
<AbstractNode.png>
<ASTNode.png>
Thank you very much for your help.
Best regards,
Youbao