Reading Website Data

If I knew the url of a website page, is there anyway to read data from the page back into SL?

For example, a Shoutcast server page contains certain information (http://66.34.54.162:8014/). This type of page has a set format. I'm just wondering if there is anyway to bring this data back into SL?

Any help appreciated.

Toby

it's primarily a llGetSubString thing..you read the webpage with llHTTPRequest and process it using llGetSubString to get the bits of that source code that you want to use in the http_response.

You have to be careful with HTML pages, because they typically contain a lot of irrelevant stuff before the actual data (such as document layout, formatting, stylesheets, and so on). As such, a lot of webpages will get cutoff, as the amount of data you can get back in an HTTP response in LSL is pretty limited. You also need to watch your script memory limits if you're doing a lot of processing on the data.

The ideal solution, if you have access to the original data, is write a script (in PHP or whatever) that just outputs the raw data, in a simple, predictable, text-only format. Or failing that, a script which 'scrapes' the original webpage externally to extract the data ready for LSL (since PHP and such like have way more resources than LSL).

EDIT: just checked, that page you linked to in the original post runs to nearly 5000 bytes, whereas your HTTP response in LSL will only give you 2048, as far as I know.

Here is a PHP script that should be useful as a filtering llHTTPRequest() proxy. If does the following:

1.) Takes a URL (the 'url' parameter) and fetches either XML contents (if the 'loadxml' parameter is set and non-zero) or HTML contents (otherwise).
2.) Optionally performs an X-Path query on it (the 'xpath' parameter, if set).
3.) Converts the result to either XML (if the 'xmlresult' parameter is set and non-zero) or text (otherwise).
4.) Optionally performs a Perl compatible regular expression match (the 'regex' parameter--if set--grabbing the nth capture group if the 'regexgroup' parameter is set to n, or the whole match if it is unset or zero).
5.) Optionally takes a sub-string of the result so far (the 'substrstart' and 'substrlen' parameters, if set).

I haven't compiled or tested this, so it may need a couple minor syntax fixes.

<?php

// parameters

$url = $_REQUEST['url'];
$loadXml = $_REQUEST['loadxml'];

$xPath = $_REQUEST['xpath'];

$xmlResult = $_REQUEST['xmlresult'];

$regex = $_REQUEST['regex'];
$regexGroup = $_REQUEST['regexgroup'];

$substrStart = $_REQUEST('substrstart');
$substrLen = $_REQUEST('substrlen');

// load the document

$response = new HttpRequest($url)->send();
$responseMajorCode = (int)($response->getStatusCode()/100);
if ($responseMajorCode != 2)
{
   HttpResponse::status($responseMajorCode);
   return;
}
$content = $response->getBody();

$document = new DOMDocument();
if ($loadXml)
{
   $document->loadXML($content);
} else
{
   $document->loadHTML($content);
}

// perform any X-Path query and convert back to string

if ($xPath)
{
   $nodeList = new DOMXPath()->query($xPath);

   $result = '';
   if ($xmlResult)
   {
      foreach ($nodeList as $node)
      {
         $result .= $document->saveXML($node);
      }
   } else
   {
      foreach ($nodeList as $node)
      {
         $result .= $node->textContent;
      }
   }
} else
{
   if ($xmlResult)
   {
      $result = $document->saveXML();
   } else
   {
      $result = $document->textContent;
   }
}

// perform any regex matching

if ($regex)
{
   if (preg_match($regex, $result, $matches))
   {
      if ($regexGroup)
      {
         $result = $matches[(int)$regexGroup];
      } else
      {
         $result = $matches[0];
      }
      if (!isset($result))
      {
         $result = '';
      }
   } else
   {
      $result = '';
   }
}

// trim to sub-string, if applicable

if ($substrStart)
{
   if ($substrLen)
   {
      $result = substr($result, (int)$substrStart, (int)$substrLen));
   } else
   {
      $result = substr($result, (int)$substrStart);
   }
} else if ($substrLen)
{
   $result = substr($result, 0, (int)$substrLen);
}

// result

echo $result;

?>