Error logging approaches - ideas/scripts???

Hi,

I'm interested in putting in an error logging system into my system and would be interested in any ideas / sample scripts. I was interested in:

* ability for me (admin) to see/get error messages that arise from the system irrespective of whether I'm online or not

* periodic email out to the admin with latest errors (if there are any)

* cater for mulitple prim systems - I'm not sure here best approach re whether all errors shoudl be trickled back to the calling/parent script for logging, or whether logging should occur directly in the prim/script it occurs and be logged

* ideally ability to see errors in chronologic order (e.g. with a prim/script name field), i.e. to cover a multi prim, multi script application

* ideally ability to in realtime adjust logging level across all scripts, or some of the scripts

Tks

I use a logging system in one of my projects; It's just a very simple dedicated script that listens for link messages of a certain type, takes the string contents, time stamps it and dumps it to a large string. It holds about 50 events before it needs to dump them (via email).

Of course, you can only log events that a running script can generate. There's absolutely no way to log for things like, say, a runtime error. Scripts that crash or otherwise are stopped by errors are effectively dead to us. No handy watchdog timers.. (But then again, no Robin and Friar Tuck style viruses...)

I guess you could compliment the error system with a periodic polling approach by a separate monitoring script (i.e. will create an error if all the scripts that should be running don't respond)

You can also have a separate object with an open listen to get error messages that the script dumps into chat. Open listens are bad, of course, but I sometimes use this technique temporarily for letting me know about errors that can't be detected in the script.

I dream of this:

try {
  llStartAnimation(anim_name);
}
catch (string error) {
  llInstantMessage(llGetOwner(), "error: " + error + " on animation: " + anim_name);
  get_next_animation();
}

Now that would really make me

Either that, or have an open listener on channel 0 check for the runtime error message.

Worthwhile test: Try having a script listen on channel 0 for messages broadcasted from the object it is in. Im not 100% sure that error messages chatted by a script can be picked up from other scripts within the same object. It would be awesome if it did work though.
==Chris

Good idea Chris, except based on recent experience I know (95% sure) that I have been having some silent runtime errors due to memory limits . I posted a bug on this to Linden Labs.

With this in mind I'll probably stick (when/if I get to this stage) to a master script that polls periodically checking system health, then restarts the system if it hangs....ummm hopefully llResetOtherScript()could be used for this.

All - do most of the major games (e.g. tringo) have a master monitor and automated reset script? I wonder how they deal with problems when they themselves are offline? I guess what I'm raising here are ways that people handle (a) system wide RESET, and (b) ability to initiate such a reset when there're offline.

I'm 95% sure that objects are deaf to any chat generated from within the prim that they're in, even if it's a different script. I'm less sure about different prims within the same object though.

Yep... I've seen this happen too. I think sometimes as a script gets very close to running out of memory, but not quite enough to push it over the limit, some scripts seem to just freeze up. Very hard thing to prove though.



Personally I would not have a listener on channel zero listening for errors. First because it would cause too much lag, and second because even if a script does error you can't do anything about it anyway. I think this is a major hole in LSL right atm. Read the wiki page for llResetOtherScript and you'll see that it will not reset a script which has crashed. AFAIK the only way to fix a crashed script is to reset it manually. I was very annoyed a while back when LL decided to make 'Reset Scripts in Selection' only work if the object is modify (although I can sorta understand why they did). So the situation we're at is that if you sell an object which is no mod, and a script crashes, the object is permanantly stuffed and would have to be replaced. Unless I'm missing something, but last time this was discussed in the forums no-one pointed out any other way around that problem.



Tringo 1.2 does not have a monitor, nor does it have a reset. The board is modify (originally so owners could change the config notecard) but thank god it was or I woulda had a lotta pissed off owners when LL changed how modify affects 'Reset Scripts in Selection'. This is one of the reasons why I'm working to make the next Tringo transferrable because atm hosts have to call the owner back to do a reset if anything goes wrong. In some cases people would like to pass the board around so the person hosting actually owns it at the time. Not a nice solution but we don't have a lotta choices in this area.

I'm also putting a reset function in the next version but that's more to stop crashes from happening in the first place, rather than recovering after one has happened. The theory is that if the main scripts are reset once in a while, hopefully any memory leaks that haven't been caught won't have long enough to build up and cause a crash. The only diff with using the scripted reset compared to 'Reset Scripts in Selection' is that the scripted one will make sure the scripts are reset in a more sensible order.

I've never tested it, but I don't think scripts can pick up that runtime error. I could be wrong, however. Part of the problem with this is that the runtime error doesn't tell you WHICH script errored out -- This really sucks if you have several scripts in a single prim, and any one could be a potential canidate.

It also really sucks that the only way to get a runtime'd script to start again is get the owner to recompile or restart it. (I can sympathize with pain of the Reset Scripts only working for those with modify permissions...) I've had to be extra-conservative with memory in cases of causing a runtime error and effectively turning the (no-mod) item into a paperweight for the future owner.

Thanks for the info Kermitt - I have some follow up questions if that is ok

Wow..this is a major flaw. Are Linden Labs well aware of this? How can we raise it to their attension/push for it to be addressed?

I'm not online at the moment but does the person have to be the owner too? (i.e. as well as the object being modify) So this is not the best approach then as (a) if the objects modify anyone can get the code [I think?], and (b) doesn't allow for an ordered start up as you mention. Is this correct? (just verifying my understanding)

So Kermitt is your approach then for the owner to open the script manually and restart. That is this way not everyone can view code, but just the owners that have bought the system?

What about right now a way to allow an administrator (non-owner) the ability to reset the system, but without them being able to see the script code, is this possible?

Thanks again

Reading Kermitt's post makes me sad that LL doesn't see what a wonderful opportunity they are wasting. They like to claim Tringo as one of the SL "success stories", which of course it is, but they don't (apparently) seize the chance to work closely with the developer to make a better development platform. Sorry folks, I don't mean to derail a useful thread with my whining. <heaves huge sigh and gets back to work>

I've bug reported it, talked about it in the forums, and even send an email to support (cc'd to Phillip) and the only response I ever got was one email back from Phillip asking if I ever got a reponse from anyone else. I responded 'no' to that and once again have heard nothing (and that was quite a few months ago. I'd have to check my emails at home to see exactly how long). I've got one other bug that's in a similar situation to this where LL have backed me into a corner with absolutely no work-arounds and I can't get any response from them about it. /111/b7/80403/1.html
That one is bad enough that it's actually making it impossible for me to release something, yet no-one at LL will even aknowledge it.



Well if they're not the owner they won't have mod rights. You can put mod rights on the object, but no-mod on the script. In that situation the owner can still reset the script, but can't see the code. Usually I get people to reset each script individually cause in most cases there's some master script that really needs to be reset last for everything to work properly, and I've yet to work out how 'Reset scripts in selection' decides what order to reset things in. There's no obvious pattern that I've been able to spot yet, but then I haven't looked at that side real closely.



You can make that possible using llResetOtherScript(), but as I said if a script crashes that won't work. So if a scripts crashed no-one but the owner will be able to fix it. Not sure if granting someone mod rights changes that. I have a feeling I've tried that b4 and it didn't work, but I'd have to test it again to be sure. (I can't get in world atm either).

Totally with u on that one Sky. I was reading a post just yesterday about the fact that LL add silly little things in upgrades that no-one really wants, and yet consistantly seem to ignore some of these really important bugs. Frankly, it's really starting to piss me off.

If you have mod rights over the owner's objects you can reset scripts in their objects just the same as they can. A tiny little bit of the permissions system that actually works as you might expect!

Sadly, even if the script is full perms in a full perms object it shows as nc, nm, nt for you even if you've got the modify rights but at least the first bit is about right!

Ooh don't get me started at how annoying this is. My partner and I have modify rights to each other's stuff but it's such a pain to see each other's scripts. Set the group, share with group, set share with group and next owner permissions on the script, set the right group active for your avatar. Click your heels together three times and say the magic words and maybe you can see the script. Aarrggh!

From the Wiki:

Q: Why doesn't my object pick up it's own spoken chat text in listen()?
A: Objects do not 'hear' themselves talk. This helps prevent infinite loops which would probably happen often if objects could hear themselves. For example, if your script returned a message saying 'Unknown command', you could go into an infinite loop easily if you didn't filter out 'Unknown command'. This applies to all of the local chat functions: llWhisper, llSay, and llShout.

You need to be careful how you interpret that. The important comment is at the end where it says....
"This applies to all of the local chat functions: llWhisper, llSay, and llShout."

I have just tested and it does not seem to apply to messages that were caused by errors. I just created an object and put in a script with an open listen on channel 0. Then I created a second script in the same object which just caused a math error by doing a divide by zero. The listen event did get raised twice when in the first script when the error happened in the second. Once for "Script run-time error", and then again for "Math Error". So you can have a script catch errors which happen in other scripts.

One thing I found very interesting here is if I made the listen more restrictive so it'd only listen to this object with either...
llListen(0, "", llGetKey(), "";
...or...
llListen(0, "", NULL_KEY, "Object";
...then it no longer caught the error messages. So that suggests that the reason the listen can catch the error messages at all, is that the error messages aren't technically coming from the object.

Anyway... once you've caught the message all you could do is notify someone. Without llResetOtherScript() being able to work on crashed scripts it's all a bit pointless.