Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Market Data Scraper (PHP)

MC Seattle
Registered User
Join date: 3 Apr 2006
Posts: 63
04-10-2006 16:36
CODE
#!/usr/bin/php
<?php

// Second Life Market Data Scraper
// v1.0.0
//
// This is an example script using curl to scrape market
// data off the secondlife.com homepage. You need the curl
// library compiled with OpenSSL libraries, and write
// permissions to a cookies.txt file in the same directory
// as the script. You can accomplish this by creating an
// empty cookies.txt file and adding group write permission
// in most cases. You will probably want to modify the
// script to do something more useful than printing the
// variables out, like storing them in a database and
// running the script from a crontab once a day.

$url = "https://secondlife.com/account/login.php";
$username = "Firstname";
$lastname = "Lastname";
$password = "password";

/*************************************************/

function remover($string, $sep1)
{
$string = substr(strstr($string, $sep1), 1);

return $string;
}

function remover2($string, $sep1, $sep2)
{
$string = substr($string, 0, strpos($string,$sep2));
$string = substr(strstr($string, $sep1), 1);

return $string;
}

ob_start();

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); // SET URL
curl_setopt($ch, CURLOPT_TIMEOUT, 5); // TIMES OUT AFTER 6 SEC
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookies.txt');
curl_setopt($ch, CURLOPT_POSTFIELDS, "form%5Btype%5D=second-life-member&form%5Bnextpage%5D=%2Fcurrency%2Fmarket.php&" .
"form%5Bpersistent%5D=Y&form%5Busername%5D=$username&form%5Blastname%5D=$lastname&" .
"form%5Bpassword%5D=$password&submit.x=42&submit.y=1&submit=Submit");

curl_exec($ch);

if (curl_errno($ch)) {
print curl_error($ch);
exit(1);
}

$string = ob_get_contents();

curl_close($ch);
ob_end_clean();

list(,$second) = explode('<table border="0" cellpadding="2" cellspacing="0" width="240">', $string);
list($first) = explode('</table>', $second);

$lines = explode("\n", strip_tags($first));

foreach ($lines as $line) {
if (strpos($line, "volume:L$")) {
$value = str_replace(',', '', remover($line, "$"));
echo "Volume: " . $value . "<br>\n";
} else if (strpos($line, "high:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's High: " . $value . "<br>\n";
} else if (strpos($line, "low:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's Low: " . $value . "<br>\n";
} else if (strpos($line, "average:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's Average: " . $value . "<br>\n";
}
}

?>
Nada Epoch
The Librarian
Join date: 4 Nov 2002
Posts: 1,423
Discussion Thread
04-13-2006 07:05
/54/a0/99999/1.html
_____________________
i've got nothing. ;)
frosty Lulu
Registered User
Join date: 30 Jul 2005
Posts: 1
Market Data Scraper - Is CURL the only way
06-23-2006 15:01
Hi.
Before I dismantle my entire working system & rebuild it for the sake of an experiment with Curl - is there any other way of doing this. I am using PHP on XAMPP for Windows XP

From: MC Seattle
CODE
#!/usr/bin/php
<?php

// Second Life Market Data Scraper
// v1.0.0
//
// This is an example script using curl to scrape market
// data off the secondlife.com homepage. You need the curl
// library compiled with OpenSSL libraries, and write
// permissions to a cookies.txt file in the same directory
// as the script. You can accomplish this by creating an
// empty cookies.txt file and adding group write permission
// in most cases. You will probably want to modify the
// script to do something more useful than printing the
// variables out, like storing them in a database and
// running the script from a crontab once a day.

$url = "https://secondlife.com/account/login.php";
$username = "Firstname";
$lastname = "Lastname";
$password = "password";

/*************************************************/

function remover($string, $sep1)
{
$string = substr(strstr($string, $sep1), 1);

return $string;
}

function remover2($string, $sep1, $sep2)
{
$string = substr($string, 0, strpos($string,$sep2));
$string = substr(strstr($string, $sep1), 1);

return $string;
}

ob_start();

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); // SET URL
curl_setopt($ch, CURLOPT_TIMEOUT, 5); // TIMES OUT AFTER 6 SEC
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookies.txt');
curl_setopt($ch, CURLOPT_POSTFIELDS, "form%5Btype%5D=second-life-member&form%5Bnextpage%5D=%2Fcurrency%2Fmarket.php&" .
"form%5Bpersistent%5D=Y&form%5Busername%5D=$username&form%5Blastname%5D=$lastname&" .
"form%5Bpassword%5D=$password&submit.x=42&submit.y=1&submit=Submit");

curl_exec($ch);

if (curl_errno($ch)) {
print curl_error($ch);
exit(1);
}

$string = ob_get_contents();

curl_close($ch);
ob_end_clean();

list(,$second) = explode('<table border="0" cellpadding="2" cellspacing="0" width="240">', $string);
list($first) = explode('</table>', $second);

$lines = explode("\n", strip_tags($first));

foreach ($lines as $line) {
if (strpos($line, "volume:L$")) {
$value = str_replace(',', '', remover($line, "$"));
echo "Volume: " . $value . "<br>\n";
} else if (strpos($line, "high:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's High: " . $value . "<br>\n";
} else if (strpos($line, "low:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's Low: " . $value . "<br>\n";
} else if (strpos($line, "average:L$")) {
$value = str_replace(',', '', remover2($line, "$", " /"));
echo "Today's Average: " . $value . "<br>\n";
}
}

?>