System Administration Approaches/Requirements (e.g. full system reset, healthcheck)

Hi all - I'm keen to put some administration capabilities into my system, as its becoming a headache without something in place. Here are my initial thougths. Does anyone have any comments / existing scripts they woudl like to throw into the mix? I've been around long enough to know it worth asking the scripting community first for ideas on LSL before diving in

ADMINISTRATION
===========
Assumptions
[1] System is a multiple prim/multiple script per prim design
[2] Comms scripts rely on prim UUID, hence handling rezz'ing is required (i.e. where prim will have a different UUID)
[3] Object Discovery is used initially to obtain UUID for object based on name (e.g. check for linked, else use SENSOR, i.e. then manually move objects > 100m after this if required)
[4] Scripts may hold UUIDs after the initialisation/object discovery phase

Requirements
[10] Strive for automatic recovery after any given script failure (i.e. avoid need for other scripts to be reset + need for manual determination of what scripts would have needed to be reset)
[11] One click rezzing of a system at a location

Scenarios to be covered:
[21] Setup/Rez of a system at a new location (strategy [32])
[22] Full System (all scripts) Reset (strategy [32])
[23] Health Check (are all components good) (ref [33])
[24] Update/Restart Single Script (e.g. during dev + for robustness) (ref [31])

Strategies
[31] Scripts publish "I've been reset" to other scripts in prim to allow them to take appropriate action (ref [24]) (Tks Christopher Omega)
Q - does this need to be sent to all objects in system?
[32] Full System Restart Capability (resets all scripts with knowledge of startup script dependencies) (ref [22])
[33] Health Check Capability (ref [23])

Design [33] - Health Check
=================
Requirements
[41] Check all scripts are running (ref [51] & [52])
[42] Check comms scripts are good (e.g. listeners established) (ref [52] & [53])
[43] Check object discoveries complete (neccessary object UUIDs are good) (ref [53])
[44] Checks scripts are in an expected state (ref [51])

Strategies
[51] All scripts implement "isScriptOnline" link message which can be called (Tks Christopher Omega)
[52] Healthcheck manager (in main prim) and remote scripts handle the propagation and consolidation of a system wide "are all scripts online" (ref [41])
[53] When scripts are answering "isScriptOnline" they should perform a test comms test to any object they need to communicate with in this

Design [31] - Single script robustness strategy
==============================
Strategy Detail
[61] Scripts publish "I've been reset" message per [31]
[62] Other local scripts with a dependancy on this script detect this and take action
Q - What if a script in object B has a dependency on this reset script in object A - perhaps the script published "I've been reset" event needs to be picked up by the local HealthCheck script (with knowledge of dependencies) so it can determine whether the event needs to be propagated to other objects. Therefore the Healthcheck scripts (with topology knowledge) would need a representative script in each object.

Design [32] - Full Restart
[71] Manager script with knowledge of script dependencies (if they exist) so it can co-ordinate the ordered reset of scripts in all objects in the system
(need to add more detail here - potential consolidation of the healthcheck manager scripts with the restart script(s))


TKs

It seems like the list is less administration than robustness and health testing. You seem to have covered the health testing rather well, just ensure that you have a dependable and bug-free script with which to reset scripts when they stop responding.

I'm assuming this is about scripts in objects that you no longer own. If not, ignore the following sentences.

On the administration side of things, perhaps a back-door script with access to the inventory commands might be in order. The most important one would be llSetRemoteScriptAccessPin (and give the end-user modify permission on the prims).

Anyways, I doubt how often such a thorough health-check is needed. LSL scripts rarely are overloaded with input.

in particular any feedback on how poeple handle "Full System Restart Capability" say after rezzing a new system would be appreciated - are people implementing a startup-up-dependancies based approach for example?

How do people normally do this can I ask? For example (guessing here):

1. package updated scripts (no mod) + script updater into an object + note_card with matrix of script-to-object_names (i.e. knowledge of which objects in the system which scripts reside in, perhaps in the case where there is a common script in an object with lots of linked prims [e.g. display] indicate this to use a different approach here?)
   
2. give object to system owner
   
3. they run the script updater which then:
   
4. - uses a SENSOR approach to find UUID for the rezzed system objects
   
5. - uses llRemoteLoadScriptPin to update scripts in the objects (or perhaps another approach for when a new script is required to be trickled through a large set of linked objects like an xytext display?)
   
6. - ensure scripts/system is restarted, using "llResetOtherScript" / "llSetScriptState". Ummm I'm not even sure what the best approach is here , may depend on nature of changes to system. Is having a system "full restart" function which the update script can trigger the best perhaps?
   

Tks - if anyone has any scripts they'd be willing to include that would be great

The backdoor idea was really only to let you modify systems which are custom jobs-- for example, someone wants you to add a little feature to or tweak the system, and it would be nice if you can do it transparently without them having to do anything.