Testing if rotation is within a cone, and move to edge

This is a pretty complex one. What I want to do is, given a rotation, test to see if it lies within a given infinite-length cone as described by another rotation, plus an angle describing how far from the rotation the cone's edges are.

If after testing the rotation I find that it is within the cone, I want to be able to take it and calculate a new rotation that lies along the nearest edge of the cone.

So, for example; say I have a cone that is pointed straight upwards, with a 45º angle describing the 'size' of it. I am given a rotation that lies within the Western side of the cone, I should be able to test that the rotation is in fact within the cone, and then calculate a new rotation that lies exactly on the Western 'edge' of the cone.

My intended use is in camera control, as I have a rotation and a distance that describes the camera's position. The position is chosen using an algorithm, however it can end up being placed directly above or below an object which I don't want, as camera behaviour is a bit odd in these places as it will 'flip' and things like that, so I want to exclude these by pushing the camera away.

What are you saying? I don't get it Couldn't you rephrase it please

Okay uhm...one way you can describe a cone is with a rotation and an angle:
The rotation describes the centre-line of the cone originating at <0, 0, 0> and extending an infinite distance. A vector could be used, but I have a rotation, llRot2Fwd() would get the vector describing the same line.
The angle describes the "arc" of the cone, so, if you took a slice of the cone, it would describe the angle (in radians) from the central line, to any point on the circumference of the circular slice of cone you're looking at.

If there's an easier way to define the cone then please let me know. The above is basically how llSensor() works, except the rotation is always <0, 0, 0, 1>, so you only have to define the arc.

With the cone defined, I want to be able to take a rotation (again describing an infinitely long line originating at <0, 0, 0> and test whether the line it represents lies within the cone. If it does, then I want to be able to calculate the nearest rotation which lies on the edge (circumference) of the cone. So I'm looking for two things.

I've come up with way to test for the first, though I'm lacking a test for directionality, which in my case isn't actually a problem, but means it isn't technically complete as this will return TRUE if the rotation lies within a cone pointing the opposite way.

integer isRotWithinCone(rotation coneRot, float coneArc, rotation test) {
    return llAngleBetween(coneRot, test) <= coneArc;
}

In order for the above to be complete, could someone describe the easiest way to test that the rotations are pointed in the same general direction?

This doesn't make sense for a rotation, but assuming you mean, say, the direction that the "forward" (local x-axis) direction of an object is pointing, and assuming that you mean a right circular cone, the following logic should work. You could use the second function to rotate the whole prim/object, using llRot2Fwd(llGetRot()) as the 'vec' parameter.

This code has not been compiled or tested, so it may need some fixes. Hopefully it'll provide at least useful pseudo-code though.

// Tests whether a vector direction is within a cone (the vector direction
// and the cone's properties are defined in the same coordinate frame).  This
// function assumes you are interested only in the "positive" half of the cone.
// It should be a simple matter to extend it to a double-sided cone if desired.
//    param coneAxis - A unit vector pointing in the direction of the cone's
//       axis of symmetry.
//    param coneAngle - Angle from the cone's axis of symmetry to the boundary
//       surface of the cone, in radians.
//    param vec - The vector to test.  This can be thought of as an offset
//       from the cone's origin.
//    returns - A true value iff vec lies on or in the cone.
integer isVecInOrOnCone(vector coneAxis, float coneAngle, vector vec)
{
   return coneAxis*llVecNorm(vec) >= llCos(coneAngle);
}

// Calculates a transformation that will rotate a vector to lie on the surface
// of a cone at the nearest point (the vector direction and the cone's
// properties are defined in the same coordinate frame).  This function assumes
// you are interested only in the "positive" half of the cone.  It should be a
// simple matter to extend it to a double-sided cone if desired.
//    param coneAxis - A unit vector pointing in the direction of the cone's
//       axis of symmetry.
//    param coneAngle - Angle from the cone's axis of symmetry to the boundary
//       surface of the cone, in radians.
//    param vec - The vector to test.  This can be thought of as an offset
//       from the cone's origin.
//    returns - A rotation that will transform vec to the nearest point on the
//       surface of the cone that has the same magnitude as vec.
rotation rotationOfVectorOntoCone(vector coneAxis, float coneAngle, vector vec)
{
   vector axisOfRot;
   float angleOfRot;

   vector unitVec = llVecNorm(vec);
   if (llFabs(coneAxis*unitVec) > 0.999)
   {
      // Choose arbitrary axis
      if (coneAxis.x > 0.999)
      {
         axisOfRot = <0.0, 0.0, 1.0>;
      } else
      {
         axisOfRot = llVecNorm(<1.0, 0.0, 0.0>%coneAxis);
      }
      angleOfRot = coneAngle;
   } else
   {
      axisOfRot = llVecNorm(coneAxis%unitVec);
      angleOfRot = coneAngle-llAcos(coneAxis*unitVec);
   }

   return llAxisAngle2Rot(axisOfRot, angleOfRot);
}

// Rotates a vector to lie on the surface of a cone at the nearest point (the
// vector direction and the cone's properties are defined in the same
// coordinate frame).  This function assumes you are interested only in the
// "positive" half of the cone.  It should be a simple matter to extend it to
// a double-sided cone if desired.
//    param coneAxis - A unit vector pointing in the direction of the cone's
//       axis of symmetry.
//    param coneAngle - Angle from the cone's axis of symmetry to the boundary
//       surface of the cone, in radians.
//    param vec - The vector to test.  This can be thought of as an offset
//       from the cone's origin.
//    returns - A new vector with the same magnitude as vec that lies on the
//       surface of the cone.
vector rotatedVectorOnCone(vector coneAxis, float coneAngle, vector vec)
{
   return vec*rotationOfVectorOntoCone(coneAxis, coneAngle, vec);
}

Thanks for the code! No time to test yet but it looks fine, except you use the name coneAxis twice in the first function, I assume the second occurrence (the float) is meant to be coneAngle

The reason I'm using rotations is because I need them for later manipulations, so following on from that since you're assuming use of the forward-axis (which are the ones I'm interested in anyway), how would I convert back into a rotation? Is there an easy-way to do that with a forward-axis direction-vector?

exactly the same as you'd do for any other vector --> rotation, use llEuler2Rot. direction doesnt mater so much as arc, you'll want to test against the undirected angle from the centerline of the forward cone.

Right you are. I fixed the variable name above. Here are some convenience functions that take and return rotations instead of vectors:

// Calculates a transformation that will rotate a local forward vector to lie
// on the surface of a cone at the nearest point (the rotation and the
// cone's properties are defined in the same coordinate frame).  This function
// assumes you are interested only in the "positive" half of the cone.  It
// should be a simple matter to extend it to a double-sided cone if desired.
//    param coneAxis - A unit vector pointing in the direction of the cone's
//       axis of symmetry.
//    param coneAngle - Angle from the cone's axis of symmetry to the boundary
//       surface of the cone, in radians.
//    param rot - The rotation whose forward vector to test.
//    returns - A rotation that will transform rot's forward vector to the
//       nearest point on the surface of the cone.
rotation rotationOfFwdVectorOntoCone(vector coneAxis, float coneAngle, rotation rot)
{
   return rotationOfVectorOntoCone(llRot2Fwd(rot));
}

// Rotates a local foward axis to lie on the surface of a cone at the nearest
// point (the rotation and the cone's properties are defined in the same
// coordinate frame).  This function assumes you are interested only in the
// "positive" half of the cone.  It should be a simple matter to extend it to
// a double-sided cone if desired.
//    param coneAxis - A unit vector pointing in the direction of the cone's
//       axis of symmetry.
//    param coneAngle - Angle from the cone's axis of symmetry to the boundary
//       surface of the cone, in radians.
//    param vec - The vector to test.  This can be thought of as an offset
//       from the cone's origin.
//    returns - A new vector with the same magnitude as vec that lies on the
//       surface of the cone.
//    returns - A rotation that is equal to rot but rotated so that its forward
//       axis is on the surface of the cone.
rotation rotatedOrientationWFwdOnCone(vector coneAxis, float coneAngle, rotation rot)
{
   return rot*rotationOfVectorOntoCone(llRot2Fwd(rot));
}