Weather Script - String Manipulation

I'm trying, and perhaps futilely, to derive weather data,
just temp/dewpoint/barometer, from METAR data which is text data
sent from airports around the world to NOAA's server where anyone
call download.

A typical string looks like:

2007/09/14 07:00 RCSS 140700Z 09009KT 9999 FEW028 32/18 Q1003 WS RWY10 NOSIG

http://weather.noaa.gov/pub/data/observations/metar/stations/RCSS.TXT

or

2007/09/14 06:50 KLAX 140650Z 15004KT 10SM CLR 17/13 A2990 RMK AO2 SLP124 T01720133

http://weather.noaa.gov/pub/data/observations/metar/stations/KLAX.TXT

* * *

Some stations provide more information than others, so the strings vary.
That's the problem. But the bit I need is always together and always
in the string.

In the first example above the bit is: 32/18 Q1008 (temp/dewpoint) barometer.
In the second the bit is: 17/13 A2990.

Is there any way to reliably pull those numbers from the string
with lsl?

I can achieve this very easily via a php script on a server
in between NOAA and SL...but I'm trying to avoid that.

And I can achieve it with a known sample with llGetSubString.

I'm not going to be surprised if there's no way.

-- Infrared

Well - as long as you know where to find the part that you need, you can split the String into a list and then extract the portions you need...

list weather = llParseString2List(data, [" "], [""]);

in the first example, you'd find your data at position 7 and 8

string temp_dew = llList2String(weather, 7); // 32/18
string bar = llList2String(weather, ; // Q1003

You could also look for the barometer-id (I guess that's what the Q1003 is?) and then the temp/dew information will be at bar-pos - 1

integer bar_pos = llListFindList(weather, ["Q1003"]);

string temp_dew = llList2String(weather, bar_pos - 1);
string bar = llList2String(weather, bar_pos);

HTH

Hi Haruki --

These are excellent suggestions...your second suggestion of using llListFindList
is probably going to help me out.

Thanks for providing some excellent code examples; I'm fairly new to lsl
programming and they are VERY clear and helpful.

- Infrared

Hi Haruki --

In further testing I've learned: you first good example is not reliable
because the string varies.

And in the second, which held promise, relies on the definition of
the string being exactly what is it. So can't match on a partial. For
example:

integer bar_pos = llListFindList(weather, ["Q1003"]);

That has to match the whole bit. But this number changes...
Q1005, Q1110...

Apparently wildcardness not possible...

We can't simply:

integer bar_pos = llListFindList(weather, ["Q"]);

for a match...it has to match the exact same string...

meh...

Infrared--

In order for a "search" type operation to work successfully, there needs to be some consistency within the string that you can be relied upon. In order to find a reliable pattern, you need to capture more samples and get all the possible arrangements that you may see.

In the two examples above, both have their 'temp/dew barometer' values in the same space-delimited word position. You mentioned additonal info can be contained, depending on the station supplying the information, so most likely this pattern isn't reliable.

More samples are needed to find a reliable pattern:

1) Are the 'temp/dew barometer' values at the same space-delimited position?

2) Does the 'temp/dew' values always immediately precede the 'barometer' value?

3) Does the 'temp/dew' values always appear together with the '/' in between?

4) Is the 'temp/dew' pattern the only other parameters, besides the date value, that contain the '/'

5) Does the date value always appear as the first parameter of the string.

If the answer is YES to item (1), the process can be achieved easily using Haruki's first example.

If the answers to items (2-5) are YES, then the process can also be achieved (just some more code).

Even if the answers are NO to some of the above, having a full (or larger set) of samples may reveal a consistent pattern that could be used to reliably extract the temp, dew and barometer values.

You could loop through the list and search for anything with a Q in it using this:

for(i=0;i<llGetListLength(weather),i++)
{
string token = llList2String(weather,i);
if(llSubStringIndex(token, "Q" != -1)
{
//might be a barometer pressure
}
}

But its kinda clumsy.. be much better if you had more definite patterns to match.

I had a look at about 20 of these Files...

The «Q» is «mostly» the first character of the bar-pressure - but sadly, not always (for example in the CBBC.TXT-File, the string looks like this: 2007/09/14 19:00
CBBC 141900Z AUTO 11002KT 9SM OVC007 13/10 A3007 RMK SLP185).

Bummer...

But it seems that the temp/dew is - besides the date - the only value that has a «/» in it and it seems, that the bar-pressure is always the value after the temp/dew...

So - as Trevor stated - you could look for the «/» character in every list-item but the first one (the date).

// Strip the first value of the list -> the date
// If you need the date, preserve it with «string date = llList2String(weather, 0)»

weather = llDeleteSubList(weather, 0, 0); 

find_pos = llListFindList(weather, ["/"]);

string temp_dew = llList2String(weather, find_pos);
string bar = llList2String(weather, find_pos + 1);

This will work, as long as there's no other value that contains a «/» and the temp/dew-values are always paired like this.

and right now, I found the CERM.TXT where I found the values «14019/1809»

you can add a little logical function to determine, whether the values are likely to be temperatures (it looks like they're in degrees Celsius, so they won't be over 50)

// Determine, whether the temperature is possible

integer max_temp = 50; // We assume, the highest value for temp is 50

list dummy = llParseString2List(temp_dew, ["/"], []);

if(llList2Integer(dummy, 0) > max_temp)  // if the extracted temp is over 50
{
// Delete the unlike value from the list
weather = llDeleteSubList(weather, find_pos, find_pos);

// Do the find again

find_pos = llListFindList(weather, ["/"]);

string temp_dew = llList2String(weather, find_pos);
string bar = llList2String(weather, find_pos + 1);

}

BUT: in the CERM.TXT I couldn't find a value that looks like the bar-pressure...
Actually, it seems all the C....TXT-Files do not contain a bar-pressure...?
And sometimes I couldn't find any valid-looking values at all...

So - all this might fail from time to time, but should work in most cases though...

HTH

ok... another one...

I found out, that there's a decoded version of this data as well...
It can be found in «http://weather.noaa.gov/pub/data/observations/metar/decoded/», the Stations-IDs remain the same.

The Pattern looks like this:

Station name not available
Jun 28, 2007 - 10:00 PM EDT / 2007.06.29 0200 UTC
Wind: from the WNW (300 degrees) at 6 MPH (5 KT):0
Visibility: greater than 7 mile(s):0
Sky conditions: partly cloudy
Temperature: 71 F (22 C)
Dew Point: 66 F (19 C)
Relative Humidity: 83%
Pressure (altimeter): 29.97 in. Hg (1015 hPa)
ob: CBYD 290200Z 300005KT 9999 SCT011 S METAR GVAC 290200Z 04011KT 9999 SCT015 22/19 Q1015 NOSIG
cycle: 2

I'd make a list of lines out of this first (although, I'm not quite sure whether this works):

I assume, you retrieve everything with the HTTP-Command and the result is stored in the variable «body»...

I hope that you can divide the body into lines by splitting it at the newline «\n» character (readers - does this work?)

list lines = llParseString2List(body, ["\n"], []);
list dummy; // a dummy-list to store extracted values

// We need another dummy lists to split the sub-data and a dummy-string

list dummy2;
string dummy3;

// put the list length in a variable, so you don't have to compute it only once

integer list_length = llGetListLength(lines); 

integer i;

integer temperature_f; // Temperature in Farenheit
integer temperature_c; // Temperature in Celsius
integer dew_point_f; // Dew Point in Farenheit
integer dew_point_c; // Dew Point in Celsius
float pressure_hg; // Pressure in Hg
integer presser_hpa; // Pressure in HPa

for(i=0;i<list_length;i++)
{
// Split the actual line at the «:»
dummy = llParseString2List(llList2String(lines, i), [":"], []);

// If it is a value that we need, process it further

// Temperature

if(llList2String(dummy, 0) == "Temperature")
{
dummy2 = llParseString2List(llList2String(dummy, 1), [" "], []);

// Now we have an array (list)  with the temperature in F at the first position

temperature_f = llList2Integer(dummy2, 0);

// For the Temperature in Celsius, we need to get rid of the brackets
// The «(nn C)»*is a the last position of the dummy2-List

dummy3 = llDeleteSubString(llList2String(dummy2, llGetListLenght(dummy2) - 1));

// delete the «C)» at the end

temperature_c = (integer)llDeleteSubString(dummy3, llStringLength -4, llStringLength - 1);

}


// Dew Point

if(llList2String(dummy, 0) == "Dew Point")
{
dummy2 = llParseString2List(llList2String(dummy, 1), [" "], []);

// Now we have an array (list)  with the dew point in F at the first position

dew_point_f = llList2Integer(dummy2, 0);

// For the dew point in Celsius, we need to get rid of the brackets
// The «(nn C)»*is a the last position of the dummy2-List

dummy3 = llDeleteSubString(llList2String(dummy2, llGetListLenght(dummy2) - 1));

// delete the «C)» at the end

dew_point_c = (integer)llDeleteSubString(dummy3, llStringLength -4, llStringLength - 1));

}

// And finally the Pressure... 

if(llList2String(dummy, 0) == "Pressure (altimeter)")
{
dummy2 = llParseString2List(llList2String(dummy, 1), [" "], []);

// Now we have an array (list)  with the Pressure in Hg at the first position

pressure_hg = llList2Integer(dummy2, 0);

// For the Temperature in Celsius, we need to get rid of the brackets
// The «(nn C)»*is a the last position of the dummy2-List

dummy3 = llDeleteSubString(llList2String(dummy2, llGetListLenght(dummy2) - 1));

// delete the «C)» at the end

pressure_hpa = (integer)llDeleteSubString(dummy3, llStringLength -6, llStringLength - 1));

}

}

I hope I made all the String-computations right... The code is not tested, so you might run into some syntax-errors 'n stuff...

But with this, you can virtually extract everything from these text-files (you might also extract the visibility, for example).

I noticed, that there are files, where there's no Pressure noted - so make sure, you have some error-handling when there's no Pressure (whatever you do with these values)...

HTH

/me points at http://www.ncdc.noaa.gov/oa/climate/conversion/swometardecoder.html , which may or may not help.

edit: better source? http://en.wikipedia.org/wiki/METAR

I guess, the provided link from Meade provides way too much information to be handled by the HTTP-Request (ony 2048 bytes). I'd go with the raw-files, means, the decoded ones...

You can also use the last parameter llParseString2List to split out significant strings such as "/", " Q", or " A" (note the leading spaces) into separate list entries, then you can find them by literal match. Since we want to keep leading spaces, we also have to keep the other separator spaces, but that can be sorted out once the entries are found. Here's some sample code:

// TEST DATA
        string data = "2007/09/14 07:00 RCSS 140700Z 09009KT 9999 FEW028 32/18 Q1003 WS RWY10 NOSIG";
//        string data = "2007/09/14 06:50 KLAX 140650Z 15004KT 10SM CLR 17/13 A2990 RMK AO2 SLP124 T01720133 ";


default
{
    state_entry()
    {

        list datalist = llParseString2List(data, [""], ["/", " Q", " A", " "]);
        
        //find the first " " separator so skip the first entry
        integer firstsep = llListFindList(datalist, [" "]);
        
        // find the temp separator "/" (past the first entru, date)
        integer indexSlash = firstsep + llListFindList(llList2List(datalist, firstsep, -1), ["/"]);

        if (indexSlash < firstsep) // compare to firstsep instead of 0
        {
            llOwnerSay("Temp Data Marker not found");
            return;
        }
        
        // find barometer
        integer indexB = llListFindList(datalist, [" Q"]);
        if (indexB < 0)
            indexB = llListFindList(datalist, [" A"]);
        if (indexB < 0)
            indexB = indexSlash + 2;    // assumes barometer follows temps, point to " "
        
        string temperatureC = llList2String(datalist, indexSlash - 1);
        string dewpointC    = llList2String(datalist, indexSlash + 1);
        string pressurehPa  = llList2String(datalist, indexB + 1);
        
        
        llOwnerSay("Temperature: " + temperatureC + "C");
        llOwnerSay("Dew Point:   " + dewpointC    + "C");
        llOwnerSay("Pressure:    " + pressurehPa  + "hPa");        
    }
}

Wow...I'm overwhelmed by all the help! From Haruki Watanabe,
Meade Paravane, Shadow Subagja, Trevor Langdon,
and Boss Spectre.

I've learned a lot.

Boss Spectre, you have done it! That code example works perfectly
and it's all I need to gather the weather data!

A little background: months ago I went looking for an analog
temp/humdity/barometer in SL and found a nice one by James
Udal. But James' only handles input from US zip codes. So
that started the journey to make a weather station (3 gauges)
that could handle international weather stats.

The script community here is so great.

Thank you.

- Infrared