LSL bytecode auto-optimization

I've been learning about the LSL bytecode for the past couple weeks and wrote some software to work with it. At first I just wanted to write an assembler, but as far as I can tell it turned out you can't really do anything esepecially awesome with it. So I got into a bytecode optimization project, and that is what I want to share

I hope this is okay in the Scripting Tips forum... I started to put it in New Products, but then I wouldn't be able to see any feedback

Background:

LL's LSL compiler produces alot of fluff code, sometimes larger instructions when there are clearly smaller alternatives, and often code that doesn't do anything but take up space. Some good examples crop up around function calls. Before calling a function, the calling routine needs to push its arguments onto the stack, and then create space on the stack for the function's local variables.

For creating space on the stack for local variables, the LSL compiler produces a PUSHARGE instruction that takes one argument indicating how many zero bytes to push onto the stack. If the function it's calling will use one (32 bit) integer local variable, the instruction would be PUSHARGE 4. For built-in functions, it will always be PUSHARGE 0, meaning *do nothing*, but taking up 5 bytes in your code.

After pushing arguments and space for local variables onto the stack, the calling routine needs to adjust BP so it points to the function's arguments and local variables, by adding their size to SP. If the function takes no arguments and uses no local variables (or is a builtin function) the LSL compiler produces code to add 0, again doing nothing but wasting 7 bytes every single time you call a builtin function.



Lsotool is the .NET app I've been working on that features the size optimization doo-hickey (and an assembler and disassembler, but they need alot of work). Currently the optimizer only does a find/replace technique, performing very obvious optimizations, but it can shave hundreds of bytes from small scripts, often over 1000 bytes on larger scripts.

To make use of it, I've rigged up this copy of slproxy to automatically optimize scripts before uploading/saving. To use it, just start slproxy and start SecondLife with -loginuri http://localhost:8080/
I included slightly more detailed instructions in the download:

I've tested the optimizer with alot of complex scripts and it seems to be dandy, but I can't make any 100% guarantees that it doesn't have any bugs.

The future of this project:

As I understand it, this project has no future. I hear compilation will be moved server-side, at which point we'll be stuck with whatever compiler LL decides to write. On the bright side, I think mono is going to make things alot better in the future.

There are a few more things I want to add to the optimizer though.

If you do decide to use this to free up space in a memory-hungry script, keep in mind that if compilation goes server-side, if you need to make changes to your script you won't be able to take advantage of bytecode optimizations. I expect the optimized script will continue running, you just won't be able to make changes without losing the magic.

Srsly gotta give credit where credit is due to Strife Onizuka for reverse-engineering LSO and writing these wiki pages:

Nice work. Sad to hear how wasteful the built-in compiler is, but doesn't surprise me in the least. I'll definetly be giving this a try.

much cool, I've been using many of strifes micro optimizations mostly as I find them this would take it a step further killing all the junk from empty function calls. Sad that it may become useless soon =/

A quick and easy optimization:

push x
push x
push 1
add/sub int,int (could also be float,float or float,int or int,float)
store x
pop
pop

goes to:

push x
push 1
add/sub int,int
store x
pop

depending how anal retentive you wanted to be, you could replace
push 1
add int,int

with:
bitnot
neg int

and this:
push 1
sub int,int

with:
neg int
bitnot

Its sad that the built in compiler is so inefficient, but then again so is the viewer. Well, at least we'll be able to make use of this in the mean time. Thanks again.

Keep in mind that LL is working on getting MONO working properly.

You can check it out on the beta grid.

Mono compiles to CIL and does so on the SERVER side. Which means that the client side compiler will at some point become deprecated and removed.

Remember how C++ was originally a preprocessor that produce C for the C compiler?

This seems like a reasonable approach for LSL. Define a client side language that has features that LSL needs, and then preprocess it into legal LSL. Some people already do this in some ways, such as adding #include directives.

I tried using this but can't seem to log in to SL. I keep getting a "Despite our best efforts ..." message. I'm wondering if there are other things I must do to make this work.

I first started SLProxy.exe and can see it running as a process on my computer.

Next I start SL using "C:\Program Files\SecondLife\SecondLife.exe" -loginuri http://localhost:8080/ -login Monica Balut (My PW) in a command file.

What more should I have to do?

It's too bad the built-in doesn't have a byte-code optimization stage. That doesn't mean the compiler's bad, it just means there's another part we wish they'd have added.

Things should be much better with Mono.

Things should be better with Mono but there are still some things that can be done to the compiler that should speed things up.

What about source code optimizations, such as loop unrolling?

Way cool, BTW. However:

Line numbers in error messages are high by 1 (as though the first line is line zero).

Variable declarations with initializations don't work: For example,

causes the following error:


I got a different error for an initialized string variable declaration. (That happened to be on the first line, though.)

I don't know why you call this "source code optimizations", since most optimizers do it at the byte code level. In any case, it's a much bigger job than this simple but effective pattern replacement tool.

Also, it seems to object to functions. A file containing only the following:

gives an error for the first line.

I'm pleased that it struck your interest

@Strife: Those are awesome indeed! I added the add/sub 1 ones and committed it to the source code repo, and the i++ to ++i one is waiting until I can add a more "fuzzy" find/replacer, at which point I also want to add some stuff that makes use of DUP and such.

I was trying to hold off replying until I could get an updated download ready, but oh well... I'll reply again later ;0

@Lee: A preprocessor's a neat idea we could still do once compilation is done on the sim, I dunno how much optimization would be possible but #includes and stuff would be cool {:

@Monica: It seems like you should have it right. "Despite our best efforts" is the message I usually get if I forget to run slproxy (I'm married to the thing). I wasn't aware there was a -login parameter, I've never used it. Worth a try: I think one person was required to use http://127.0.0.1:8080/ instead of localhost before it would work for her... Not sure.
Here's the target of the shortcut I use to launch SL:



@Lear: Line numbers do start with 0, I'll add that to the list of issues with the assembler (Edit: Fixed in SVN)
Global variables may be initialized when declared, since they can have an initialized value in the binary without requiring any code to perform the initialization; local variables on the other hand are just there so you can reserve space on the stack and use them to reference it.. to assign to a local variable you'd need to do something like this



or



And indeed, the assembler doesn't support user-defined functions yet. Combined with the fact that you gotta declare labels above where they're used, the assembler can be frustrating to try to use at this point I'm tickled that you wanted to use it though man! Check out the list of issues, mostly with the assembler

Be back later <3

Mono is currently horrendously broken, and I would not want to hang my hopes on the idea that it will be finished and ready for prime time within the next 12 months.

It's an awesome project with fantastic potential, and I'm very much looking forward to it, but it's very far from done.

I suggest that rather than finding more optimizations, make it support LSL. It's not very useful if it only supports a limited subset of the language. Functions are very important, especially when trying to minimize space usage.

BTW, above you seem to state that initialized globals should work, but they don't. I may be misinterpreting your text, but I don't see it listed as a bug.

Thanks!

Hmm, I should explain better. lsotool is like a multi-purpose thing, a swiss army knife for LSL bytecode. It can load in a compiled binary and disassemble or optimize it, two things which are working very well. Separately, there's the assembler, which is a piece of crap But it's not needed to use the optimizer.

I don't really recommend using the assembler in its current form (and don't look at the code for it unless you want to be sick), but if you really are interested, I posted a few examples that do successfully assemble: I'm not so sure I'll finish the assembler, especially since I hear Strife may be working on one and he probably sucks alot less than me at writing a compiler But I want to make sure everyone understands the assembler and the optimizer are completely separate, and the foundations the optimizer is running on are feature-complete.

To use the optimizer with your LSL scripts, the link you want is

At present I'm working on adding code to my decompiler so that it can optimize bytecode, the compiler has been shifted to the back burner... again. I'm going to do this with a bytecode tree and a search syntax similar to XPath (I'm actually tempted to convert the bytecode to xml and then just use xpath). It will probably share some classes with the compiler.

How can I run the optimizer on LSL source code?

Alternatively, if I have LSL source code, how can I get the bytecode for it, to run the optimizer alone?

Thanks

Day: I got it working. What I didn't realize was the the SLProxy died if I closed out the command box. I was expecting it to run in the background. If I keep the command window active, SL starts fine.

I agree with the need to handle functions. It's rare to never that I write a script that does not include functions.

Are you guys really saying you want to use the assembler?

The optimizer will work fine, nay, dandy with LSL scripts that have functions ._.

Thanks for the clarification! Looking forward to trying it inworld. I had the impression I could work offline from source, but evidently not for arbitary LSL code.

Monica, the part that doesn't handle functions is when you run "lsotool -a" to compile ("assemble" according to Day) LSL source into bytecode. If you're using SLProxy, should be no problem because the SL client does the compilation into bytecode, not lsotool.

However, a cohort trying to use this with my code (which compiles in LSL) said it wouldn't work using the optimizer, whereas an earlier version would. So, there may be some LSL code that generates byte code that the optimizer chokes on. I'll be sure to report if there is -- or would you prefer an IM, Day?

Oooh, if you have a piece of code that breaks it, let me know!

Otherwise, I hope I can do these things in the next few days:
* Fuzzy find/replace so more optimizations can be done
* A setting in the proxy to keep a local copy of code
* A command in the proxy to upload a new script from a file
* Think of an elegant way to put the slproxy source up in the same repository