Ok, assuming you're starting with the 3 vertex vectors, A, B and C, pre-calculate the lengths of the sides ( floats AB, BC and AC ) and the angles at each vertex ( floats aA, aB, aC ).
float AB = llDistanceBetween( A, B );
float BC = llDistanceBetween( B, C );
float AC = llDistanceBetween( A, C );
float aA = llAngleBetween( B - A, C - A );
float aB = llAngleBetween( A - B, C - B );
float aC = llAngelBetween( A - C, B - C );
I will assume, for the sake of consistency, that your diagram uses the prim's local X (width) and Z (height) axes, pointing to the right and up, repectively.
From: someone
1. how to calculate the position vector of point F
Find the midpoint of AB (E, in your diagram) just as you would find the average of two numbers, by adding the vectors together and dividing by 2.
vector E = ( A + B ) / 2;
Then add half of vector DC. AD is the length of AC times the cosine of angle A:
float AD = AC * llCos( aA );
vector D = A + ( B - A ) * ( AD / AB );
vector F = E + ( C - D ) / 2;
Note, you will want to first confirm that aA and aB are both <= 90 degrees (PI_BY_TWO radians), since these angles cannot exceed a right angle by sheering a prim. In other words, if there is an obtuse angle, it will have to be angle C, for the purposes of this example.
Also note, point F will fall on the plane of this polygon, but will not be the center of the prim. You will need to offset it by half the prim's Y scale. More on that below.
From: someone
2. how to calculate the desired rotation so the face is aligned with these three points.
llAxes2Rot is perfect for this. Input 3 vectors which point in the prim's desired local X, Y and Z directions, and it returns the appropriate rotation.
The X (fwd) vector would be B - A. The Z (up) vector would be C - D, and the Y (left) vector is the cross product of these two.
vector Yaxis = llVecNorm( (B - A) % (C - D) );
rotation R = llAxes2Rot( B - A, YAxis, C - D );
I first calculated the normalized Y-axis so that this can be used to calculate the center of the prim from point F.
vector PrimScale = llGetScale();
vector Center = F + Yaxis * PrimScale.y;
From: someone
3. how to calculate the desired sheer.
Sheer is the amount of horizontal offset applied to the top (+Z) face in terms of the base's dimensions. In this case, the ratio of the lengths AD to AB.
float Sheer = ( AD / AB ) - 0.5;
0.5 is subtracted to translate the range from the "0 to 1" to "-0.5 to +0.5".
Note, however, that the sheer of a prim is represented by a low-precision number, apparently rounded off to the nearest-hundredth (though it's probably more like the nearest-128th, I haven't done any tests to be certain), so point C will almost always be slightly off, depending on where it happens to fall between A and B.
Note that all of the above isn't an optimal method, but it should generally work.