Got bored, wrote SL code transformer

<edit>
Just to keep things up to date (and avoid questions later): following some digging into the internals of the SL client and communication, I am scrubbing all SL-related projects for at least a major version of SL. And if wish the poor developers luck ever getting this mess stabilized and optimized. Work on Whistle will continue elsewhere, but I plan to focus on the optimizer and its own JIT rather than designing toward .Net or similar.
</edit>

In a fit of boredom (until I get my SL client crashes fixed, it will be a chronic state) I have finished off a scanner, parser and tree representation of LSL script. The output generation and formatter are not quite finished yet, but should be soon. In the meantime...

What syntax extensions would you like to see for LSL?

Currently I have implemented support for:

1. #listOrString -> llGetListLength(list) or llStringLength(string)
   
2. listOrString[5] -> llList2InferredType(list, 5) or llGetSubString(string, 5, 5)
   
3. listOrString[5,10] -> llList2List(list, 5, 10) or llGetSubString(string, 5, 10)
   

And am considering adding:

1. user-defined variable types (with operator overloading)
   
2. varargs/optional arguments on functions
   
3. pseudo-events triggered via link messages
   

Any suggestion that can be mapped to existing LSL syntax is great. Obviously I am fairly new to the scene, so other people may have expectations and coding styles very different from mine. As an example of an easy mapping: calling a function with optional arguments omitted just means copying the default values from the function declaration.

So what have ya got?

<edit>Messed around for a while after lunch. The parser (not symbol table or code gen) currently supports anything within anything. Interestingly classes and states are very similar (including inheritance), and there is no longer a syntactic difference between functions and states. Quick sketch of how the parser looks now:

// abstract means "this is not a real LSL state"
abstract state base {
    // Put events here for them to be available in both light_on and light_off
    // Overriding behaves as expected -- a touch event in this state would
    //    never be triggered.
    touch(integer n) {
        llOwnerSay("Never called, OR included in the generated script" +
            "Yeah for reachability analysis");
    }

    // Yup, virtual functions in states
    state_entry() { llSetColor(GetCurrentColor()); }
}
state light_on : base {
    touch(integer n) { state light_off; }
    vector GetCurrentColor() { return <1,1,1>; }
}
state light_off : base {
    touch(integer n) { state light_on; }
    vector GetCurrentColor() { return <0,0,0>; }
}

Embedding the symbol table in the tree is fairly easy. So I should have a working demo in a few days. At that point I will bounce the bison/flex source off LL legal to ensure they either agree it is not a derived work or are willing to put a BSDish license on their claim. A basic version of the translator will definitely be free, though I am still mulling the more complicated (and time-consuming) features.
</edit>

Excellent! Some ideas:

* allow variables to be declared within a state. For now, these would just become global variables (with a unique prefix so they don't name clash)

* <type> <variable> = <big-huge-expression>
This would translate to:
<type> <variable>; <variable> = <big-huge-expression>

* string interploation: support variable and expression references inside of strings like perl:
string $name ...
integer $cost ...
vector $loc ...
"hello, $name, I cost L\$$cost and am located at ${loc.x},${loc.y}."
becomes
"hello, " + (string)name + ", I cost L$" + (string)cost + " and am located at " + ((string)(loc.x)) + "," + ((string)(loc.y)) + ".";

Notices the escaping and handling of the literal dollar sign.

----

This reminds me of my LSL transforming project: I use marked up LSL code so that I can compile it under C++ (!) and run it against my (very) partial library implementation of LSL functions. That way I can run, debug and test my scripts in my nice development environment (Xcode on the Mac). Then I run the marked up LSL code through a stripper and out pops just the LSL. Obviously this only works for limited types of LSL code (I was doing some heavy massive data manipulation).

Hashtables.

>>=
<<=
>>>
>>>=
&=
|=

real arrays

Static arrays, true constants, possibly enums. These should be easy enough to add to LSL

Finished a first cut of string interp, and fired the base grammar files of to LL for their approval. It still has bugs, but is as close to the stuff Cory Linden posted in the Hotline forum as it every will be. Assuming no Copyright grumbles from them, I will get into the code generation and grammar debugging this weekend.

Latest set of tweaks (grammar and parse tree only):

1. String interpolation: Makefile-like, so `Hello $(llGetDetectedName()).` (currently keeping the double-quoted string unchanged for compatibility).
   
2. Operators: Everything and the kitchen sink. Unsigned right-shift is definitely available, and if you can think of anything to do with "#=" go ahead and overload it.
   
3. Fixed-size arrays
   
4. Still pushing around approachs for enums and bitfields.
   
5. Imports with namespaces and renaming
   
6. Mixins for classes and states, pondering single inheritance
   
7. Believe it will interpret traditional LSL code correctly
   

Constant folding will definitely happen once I get to tree transformations, other things may or may not get implemented in the first pass.

<edit>Strife: The state inheritance example is ripped from nested finite state machines (and may well wind up being written by nesting the state declarations instead of an explicit mixin). Not sure where "meaningful syntax for what is required" was aimed, and the default state was omitted for clarity. The LSL grammar restricts assignment on variable allocation to simple expressions, I agree with extending it to allow (and rewrite) more complex expressions. As mentioned above, am only interping back-quoted strings to avoid the compatibility problems. Events are being identified by name, and will probably be callable as functions -- I considered adding "function" and "event" keywords, but that confused the areas in which the new syntax was legal versus the legacy one. Have already had to force a type on functions declared in statement blocks, and may be making explicit typecasts into function invocations. Really not interesting in mucking with the LSL bytecode: will be going to LSL source for the moment and CIL later.</edit>

I don't actualy support some of the perversions suggested here. They make reading the code more difficult.

The trouble i see with the example is its a morph of function and class syntax without providing a meaningful syntax for what is required. The example lacks a default state.

Zarf's suggestion for type declaration is nieve as to the workings of LSL. LSL doesn't work quite like that. LSL sets aside memory for the value, then writes the value when it is supposedly declaired.

Zarf's suggestion for using "$" in strings would likely break existing scripts and provide something that isn't needed as it couldn't be done dynamicly (not without incuring a performance hit that would effect all scripts).

fyi: In LSL bytecode there is no difference between an event and a function syntacticly.

I personaly would like to see LSL go more towords Java or C then Perl or Python.

The suggestion for combining assignment and variable declaration is common syntactic sugar in languages. It doesn't change the underlying semantics of LSL. It's implementation doesn't alter the way variables are allocated or assigned in the underlying virtual machine. And it could be implemented in the current bytecode. An exception would be allowing this for



It wasn't clear to me that Minsk wished to create a superset language or a different language that was compilable down to LSL. I assumed the later. None-the-less, if one wants quoted strings to retain the current semantics, than interpolated strings can be signaled by an opening sigil. For example: "foo $bar" would be a literal, whereas $"foo $bar" would interpolate the value of the variable bar.

No matter how string interpolation is syntactically written it is always a dynamic operation. It is actually purely syntactic sugar for an operation you can already do. In the above example, $"foo $bar" becomes ("foo " + (string)bar) The reason this is desirable is that objects in LSL do a fair bit of formatting values for reading, and so a shorthand syntax is warranted.

There, finally got the last of the string interp parsing bugs mashed (my real work involves loooong compile times and way too much debugging). My current string interpolation syntax is below, and expands to the obvious concatenation of strings and appropriately-cast expressions (expansion is done in the lexer, so there is _really_ no overhead).

foo() {
    string a = "This is a legacy $(LSL) string with no inclusions";
    string b = `Hello $(llGetOwnerName()), this string has inclusions and \$'s`;

    string c =
`If you really $(llWantToConfuseYourself(`this $(llGetOwnerName()) works`)) too`;
} }

My aim is to allow legal LSL to be used as much as is possible. The only syntax that I have scrapped so far are typecasts: "foo)(bar)" could be either a cast to the type foo or the application of the () operator to the value foo. They will be presented as functions instead (e.g. "string(5 + 5)", which prevents the usual "string) 5 + 5" mistakes.

<edit date="Aug 7">
I won't bump this, but just in case someone peeks into this thread: Added legacy mode to the parser, which will support all existing LSL code but removes a few of the new features. Completed tree generation earlier tonight, and will be moving into code generation tomorrow. Have not heard back from LL yet, so it may be a while before a public beta is available.

Re Christopher Omega below: between operator overloading and the import statement it will be fairly easy to add the necessary operators (say in a "Java.wsl" and pull them in when needed. I am a believer in "lots of rope" programming languages, so just about anything is possible... (except changing operator precedences, I don't want to write that in the tree generation code just yet)
</edit>

<edit date="Aug 15">
Work is unfortunately requiring work, so things have been pushed back. I am tweaking the syntax a bit to be closer to C# (not too close), to simplify the CIL target later.
</edit>

Why not simply make the + string concantinator smarter? If it sees a string on its left side, it concantinates the right side to it, casting it to a string beforehand, no matter what type it is.

This way, it looks exactly like java string concantination syntax (which is very pretty and regular), even though it doesnt work the same way.
==Chris