Fastest way to remove duplicates

I have a script that compiles a large list of names (sometimes up to 40) and while doing so, needs to prevents duplicates. Here's an example of how I'm currently doing it (simplified):

list allNames = ["Joe","Jill","Tim","Tom","Sam"];
list newNames = ["Jane","Jill","John","Harris","Martin","Deena"];

integer i;
for (i=0; i < llGetListLength(newNames) - 1; ++i) {
    if (llListFindlist(allNames, [llList2String(newNames, i)]) > -1) {
        allNames += llList2String(newNames, i);
    }
}

Is there a faster way to do this? Having to check each name using a FOR loop really slows down the speed of the script.

C'mon Strife Onizuka, I know you're out there and you have a trick or two that works better.

Don't do it with two lists, do it with one. It's quicker. Check the name before adding it to the list.

sensor( integer number_detected )
{
    integer i;
    for( i = 0; i < number_detected; i++ )
    {
         string detected_name = llDetectedName( i );
         if( isNameOnList( detected_name ) == FALSE )
         {               
             allNames += detected_name;
         }
      }
}

I dunno about list speed but a for loop is usually not the fastest option, try a do-while loop.

Pad is right. That's how I do it too. The less lists in a script the better.

here is about as fast as you can get. I toyed with the idea of using a string instead of a list.
if your going to call for me... you better be prepared

list allNames;

string detected_name;
default
{
	state_entry()
	{
		llSensorRepeat("", "", AGENT, 96.0, PI, 5.0);
	}
	sensor( integer i )
	{
		do
			 if(llListFindList(allNames, [detected_name = llDetectedName(--i)]) & 0x80000000)
				 allNames += detected_name;
		while(i);
	}
}

Shouldn't this be:

list detected_name;

//...

if(llListFindList(allNames, detected_name = [llDetectedName(--i)]) & 0x80000000)
    allNames += detected_name;

probably, i've never gotten around to seeing how the += operator works when adding non lists to lists, that is if the compiler actualy puts the element in [] first. As to which is faster I've not taken the time (think i'll do that now).

Don't forget to factor in the memory costs of a list containing a string versus a string and a temporary list. Oh, shouldn't you do detected_name=""; or detected_name=[]; at the end of the sensor event?

You didn't answer an earlier question of mine, by the way, so I'll ask it again now. Why not do:

do {
  detected_name = [llDetectedName(--i)];
  if(llListFindList(allNames, detected_name) & 0x80000000)
    allNames += detected_name;
} while(i);

Which is how I would write it.

Ok I've pulled the bytecode and the compiler does not put it in [] (meaning that list += non-list, isn't the same as list += [non-list]).

Storing it as a string will save memory. As to if it's faster i still have to find out (defragmenting harddrive presently).

Thanks for the advice so far.
Let me go a little deeper into what my application does.

It's basically a sensor grid for detecting all the agents in a simulator. It then provide a means to transport directly to them. It does this by sending out four sensor probes to calculated places in the sim and each one performs a sensor. Those four sensors cover the entire surface area of the sim. Then each probe reports the data back to the host script. Because some areas of each sensor overlaps others, duplicates will be reported back and need to be filtered when compiling the list.

Because agents can be in motion, the faster the data is returned and compiled the more accurate the scan. It'll never be exact but it's already fast enough to be very useful. Now I'm just trying to squeeze every ounce of performance out of the script.

I've tried various methods to get these unlinked prims to send their data back to the host as fast as possible including:

Return to within 90m of the host and shouting each name and position.
Return to within 90m of the host and shout a set of (4) names/positions at a time.
Emailing the entire sensor data of each probe from its scan location directly to the host.

So far, it seems the first method is the fastest. It has the advantage of building an event queue for the host script to trigger as fast as events can trigger. Now it's the "checking for duplicates" that slows things down. As the list of names grows, it gets slower.

How do you retire old names from the list, Harris? How and when do you report them to the owner? Depending on how you are doing this - I would be tempted to say, get each probe's results back from the shout into list. Add the four lists together into one large list (we're sacrificing memory for speed here). Sort the list.

Now you have a sorted list, with duplicates. You are probably going to loop through it anyway when it comes times to reporting back to the owner, right? Piggy-back on top of that loop by including an if (this_name != last_name) check.

Basically, don't worry about removing dupes from the list - just filter them out of the output.

As each name/position combo is shouted it triggers a listen event. With all four probes talking at the same time, this can be in any order. At each event I check the new name against the master list. If it's not there I add it. Otherwise, I ignore it.

Names are never retired from the master list until you perform a new scan at which point the entire list is reset.

Your "add them all and then sort out duplicates later" is an interesting idea but won't work because part of my display feedback includes a total of agents detected. If there were duplicates in the list, it would report the incorrect number of agents in the sim.

Since last week, I've threaded out my program to have a dedicated script that only listens for scan results. It does it using only strings (not lists) and when finished, message links the strings to the master script which turns them into lists. This seems to have helped performance.