Welcome to the Second Life Forums Archive

These forums are CLOSED. Please visit the new forums HERE

Information on how to pull data from the secured parts of the SL website...
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 13:46
No, I'm not talking hacking, nor am I talking about invasion of anyone's privacy.

I'm talking about having software that can login to the SL website under someone's name and pull data such as his friends list.

Let me point you to the tools you need:

Visual C#, by Microsoft (or any of the other Visual Studio systems)

This bit of HTML:

CODE
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

<link href="/_styles/MAIN.css" rel="stylesheet" type="text/css">

<title>Second Life | My Account: Log In</title>
</head>

<body class="body_community">



            <table width="100%"  border="0" cellspacing="0" cellpadding="0">
              <tr>
                <td valign="top"><form action="https://secondlife.com/account/login.php" method="post">
                    <input type="hidden" name="form[type]" value="second-life-member" />
                    <input type="hidden" name="form[nextpage]" value="https://secondlife.com/account/index.php" />
                    <input type="hidden" name="form[persistent]" value="Y" />
                    <p>
                    <table  border="0" cellspacing="3" cellpadding="0">
                      <tr>
                        <td>Second Life First Name:</td>
                        <td>Second Life Last Name:</td>
                      </tr>
                      <tr>
                        <td><input type="text" name="form[username]" value="" size="20" maxlength="31" /> </td>
                        <td><input type="text" name="form[lastname]" value="" size="20" maxlength="50" /></td>
                      </tr>
                      <tr>
                        <td>Password:</td>
                        <td> </td>
                      </tr>
                      <tr>
                        <td><input type="password" name="form[password]" value="" size="20" maxlength="16" /></td>
                        <td><a href="https://secondlife.com/account/request.php">Forgot your password?</a></td>
                      </tr>
                      <tr>
                        <td colspan="2"><img src="https://secondlife.com/_img/spacer.gif" border="0" width="10" height="10"/></td>
                      </tr>
                      <tr>
                        <td><input type="image" src="https://secondlife.com/_img/buttons/b_submit.gif" name="submit" value="Submit" /></td>
                        <td align="right"> </td>
                      </tr>
                    </table>
                    <p></p>
                  </form>
                  </td>
                <td width="1"><img src="https://secondlife.com/_img/spacer.gif" border="0" width="10" height="10"/></td>
              </tr>
            </table>



</body>
</html>


Now, you will need to create a now project, and add the WebBrowser object to the form.

The WebBrowser is the key that will allow us to access the SL website and not have to worry about the issue of cookies. (remember kids, no problem is so great or evil that you can't just bypass it totally by using some utterly insane method)

You will be able to access the HTML that is displayed in the forum by use of the Object.DocumentText = "whatever" method.

This means you can set the above HTML data into the Object.DocumentText and use it to login. The object will take care of all cookies and remembering of logins.

Now all you have to do it set it up to change the destination page in the above HTML (form[nextpage]) and have it automatically open the correct page.

I have not experimented, but it may be possible to have the above HTML altered so that the data is fed in when the page is called and the send button automotically 'clicked'

Now, once it brings up the next page, you can then use that same Object.DocumentText to READ what it send, and then use parsing code to make use of it.

I will try to release a basic project that shows everything I'm talking about, But the info is already everything you need for this to work. Some things to keep in mind:

- You can always hide the object, using it to pull data without it being obvious what your doing. Also, the built in events allow you to detect the following:

DocumentCompleted: Occurs when the WebBrowser control finishes loading a document.
FileDownload: Occurs when the WebBrowser control downloads a file.
Navigated: Occurs when the WebBrowser control has navigated to a new document and has begun loading it.
Navigating: Occurs before the WebBrowser control navigates to a new document.


As you can see, you have a lot of options. Anyone here know if a web page can be set to auto post data upon entry? <hmmnnn>

Hey! Perhaps if we set the intial website address to be the POST URL?

Thoughts?
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 14:08
It would be nice to be able to extract forum posts and store them in a manner that allows for effective searching instead of the crippled search mechanism these forums offer.

Also, I want to be able to do something like store all of say, some Lindens posts, and some well known forum troublemaker's posts, and feed them into a Markov chain text generator.

Cam't somewhat the same effect be acheived by using something like "AutoIt" to login and and save the text by simulating the keystrokes or direct menu calls for File Save As?
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Magnum Serpentine
Registered User
Join date: 20 Nov 2003
Posts: 1,811
07-08-2006 14:09
I am not sure what you are trying to do but I have alerted the lindens to it.
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 14:21
From: SuezanneC Baskerville
It would be nice to be able to extract forum posts and store them in a manner that allows for effective searching instead of the crippled search mechanism these forums offer.

Also, I want to be able to do something like store all of say, some Lindens posts, and some well known forum troublemaker's posts, and feed them into a Markov chain text generator.

Cam't somewhat the same effect be acheived by using something like "AutoIt" to login and and save the text by simulating the keystrokes or direct menu calls for File Save As?


:eek:

No. Not a good idea.

Let me explain, for a search engine to work remotely, it would have to download ALL of the forums.

All 4 years of it.

In one go, more than likly.

<shivers>

No. I am NOT doing that to SL. It would be bad. Very bad. Banning bad.
_____________________
lana Birdbrain
Registered User
Join date: 17 May 2006
Posts: 66
07-08-2006 14:27
Clearly in even needing to ask this I'm showing how technologically challenged I am...... but are you talking about getting a copy of your own friends list this way or someone elses?
_____________________
Alt of Allana Dion - If I'm here, its because she was too lazy to log out and back in again.
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 14:40
From: lana Birdbrain
Clearly in even needing to ask this I'm showing how technologically challenged I am...... but are you talking about getting a copy of your own friends list this way or someone elses?



Your own.

As to if you would be willing to share it... Well, I guess it would be possible to build a system that shared such info, and would be reasonably secure. As long as you don't (FOR GOD'S SAKE DON'T :eek: ) share your password, you should be fine.
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 14:42
From: Foolish Frost
:eek:

No. Not a good idea.

Let me explain, for a search engine to work remotely, it would have to download ALL of the forums.

An URl of the form "http://forums.secondlife.com/search.php?do=finduser&u=#####" enables one to get links to posts by a certain user. The resulting page can be parsed for links to individual posts, which could then be loaded for extraction of the post bodies.

The post bodies could be stored as text files on one's hard drive and searched with grep.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 14:44
From: SuezanneC Baskerville
An URl of the form "http://forums.secondlife.com/search.php?do=finduser&u=#####" enables one to get links to posts by a certain user. The resulting page can be parsed for links to individual posts, which could then be loaded for extraction of the post bodies.

The post bodies could be stored as text files on one's hard drive and searched with grep.



I don't see what this does beyond using a forum search with keywords as well.

And it STILL does not scale. Too many people use this, and the system search engine goes BOOM!
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 14:52
From: Foolish Frost
I don't see what this does beyond using a forum search with keywords as well.

And it STILL does not scale. Too many people use this, and the system search engine goes BOOM!

Collecting the old posts locally is a one time deal. Once you have the data stored on your hard drive you can search your local copies to your hearts delight with no effect on LL's system at all.

As for the search capabilities, the forum search can't search for common words, can't search for two letter sequences, can't search for too long a phrase, can't do boolean searches; sheesh, we can't look up sin, cos, and tan in the scripting forum. All of this would be possible with locally stored copies of the posts.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 15:15
From: SuezanneC Baskerville
Collecting the old posts locally is a one time deal. Once you have the data stored on your hard drive you can search your local copies to your hearts delight with no effect on LL's system at all.

As for the search capabilities, the forum search can't search for common words, can't search for two letter sequences, can't search for too long a phrase, can't do boolean searches; sheesh, we can't look up sin, cos, and tan in the scripting forum. All of this would be possible with locally stored copies of the posts.



Good points, all. I still worry about the problem of scaling:

If you do it, no real problem. If we do it, still not really a problem.

Everyone does it, server bursts into flame and chases the lindens about the room for an hour or so...

:D

Well, not really, but I like the image, so it stands.
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 15:28
From: Foolish Frost
scaling:
Mightn't some form of p2p file sharing work for this? Distribute the entire forum database through something like Bit Torrent.

At this point I have gone enough off topic enough to stop pursuing this train of thought here.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 15:42
From: SuezanneC Baskerville
Mightn't some form of p2p file sharing work for this? Distribute the entire forum database through something like Bit Torrent.


Aye, it would be possible.

Now, considering the average forum post, the response to such a plan will be something like:

From: Havok Screamalot
OMGZ! My systemz sending data to other people without my godgiven consent written in triplicate and signed in the blood of a virgin!

From: Eggalong Noisemaker
You should SUE them over it!

From: Random Mouther
#*()$(#) you *#@#@* ()@*()#!!!! You don't know nothing! You are STUPID!

From: NobodyListens Anidoncare
It's just the information your searching from. The forum posts. Nothing secured is being sent.

From: Havok Screamalot
MY BANDWIDTH! MINEMINEMINEMINE! I hate you all and am quiting as soon as I'm finished screaming at you all!


...

(Oh... I am so going to burn for that one...)

:D

<funny enough, I know it's unfair, but I can't stop laughing when I re-read it...>
_____________________
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 15:50
From: SuezanneC Baskerville
At this point I have gone enough off topic enough to stop pursuing this train of thought here.



Nope. Feel free to pursue it. I just worry from the perspective of a sysadmin as well as a programmer. Comes from being one.

As I said, it's possible, but i worry about the stress on the system. My opinion of your sharing suggestion is that it's an overall good idea, but that each person's computer would have to be networked together, and they HATE when you talk like that. Sharing anything, information, bandwidth, manners... It seems to go against people's grain. Not sure why...
_____________________
Eddy Stryker
libsecondlife Developer
Join date: 6 Jun 2004
Posts: 353
07-08-2006 16:20
Here's a code snippet from an old project (C#). You will need libcurl.Mono, let me know if you have trouble finding the Windows binary package.

CODE
		public override ArrayList getBuyOrders()
		{
			try {
				Curl.GlobalInit((int)CURLinitFlag.CURL_GLOBAL_ALL);
				
				Easy curl = new Easy();
				
				Easy.WriteFunction curlCallback = new Easy.WriteFunction(LindeXWriteData);
				
				curl.SetOpt(CURLoption.CURLOPT_WRITEFUNCTION, curlCallback);
				curl.SetOpt(CURLoption.CURLOPT_URL, "https://secondlife.com/account/login.php");
            	curl.SetOpt(CURLoption.CURLOPT_SSL_VERIFYPEER, false);
            	curl.SetOpt(CURLoption.CURLOPT_FOLLOWLOCATION, true);
            	curl.SetOpt(CURLoption.CURLOPT_HEADER, false);
            	curl.SetOpt(CURLoption.CURLOPT_COOKIEJAR, "cookies.txt");
            	curl.SetOpt(CURLoption.CURLOPT_POSTFIELDS, "form%5Btype%5D=second-life-member&form%5B" + 
            											   "nextpage%5D=%2Fcurrency%2Fsell.php&form%5B" +
            											   "persistent%5D=Y&form%5Busername%5D=" + firstName + 
            											   "&form%5Blastname%5D=" + lastName + 
            											   "&form%5Bpassword%5D=" + password + 
            											   "&submit.x=50&submit.y=1&submit=Submit");
            	
            	curl.Perform();
            	curl.Cleanup();

            	Curl.GlobalCleanup();
			}
			catch(Exception ex) {
	            Console.WriteLine("Error: " + ex);
        	}
        	
        	ArrayList openOrders = new ArrayList();        	
        	string data = buffer.ToString();
        	data = data.Remove(0, data.IndexOf(">Open Buy Orders<"));
        	int i;
        	
        	while (data.IndexOf("bg_dashes_w_ltblue") > 0) {
        		data = data.Remove(0, data.IndexOf("\t<tr>") + 5);
        		string lineItem = data.Substring(0, data.IndexOf("</tr>"));

				////
        		i = lineItem.IndexOf(">") + 2;
        		string exchangeRate = lineItem.Substring(i, lineItem.IndexOf("<", i) - i);
        		
        		if (exchangeRate.IndexOf("$") < 0) {
					data = data.Remove(0, data.IndexOf("</table>"));
        			continue;
        		}
        		
        		exchangeRate = exchangeRate.Substring(1, exchangeRate.IndexOf(" "));

        		lineItem = lineItem.Remove(0, lineItem.IndexOf("</td>") + 5);
        		////
        		
        		////
        		//i = lineItem.IndexOf(">") + 1;
        		//string buyers = lineItem.Substring(i, lineItem.IndexOf("<", i) - i);

				lineItem = lineItem.Remove(0, lineItem.IndexOf("</td>") + 5);
        		////
        		
        		////
        		i = lineItem.IndexOf(">") + 1;
        		string volume = lineItem.Substring(i, lineItem.IndexOf("<", i) - i);
        		volume = volume.Remove(0, 2);
        		volume = volume.Replace(",", "");
        		////
        		
        		data = data.Remove(0, data.IndexOf("</table>"));
        		
        		Linden linden = new Linden();
        		linden.rate = double.Parse(exchangeRate);
        		linden.volume = int.Parse(volume);
        		openOrders.Add(linden);
        	}
			
			return openOrders;
		}
_____________________
http://www.libsecondlife.org

From: someone
Evidently in the future our political skirmishes will be fought with push weapons and dancing pantless men. -- Artemis Fate
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 16:55
From: Eddy Stryker
Here's a code snippet from an old project (C#). You will need libcurl.Mono, let me know if you have trouble finding the Windows binary package.




Why thank you! You're willingness to share is a bright moment here!

Does it need a binary, or does it install as pure functions? I abandoned the other due to it being extra DLLs that were hard to control...

And they crashed...
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 17:00
Would the free Visual Studio C# 2005 Express Edition available at http://msdn.microsoft.com/vstudio/express/visualcsharp/ work alright?
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-08-2006 17:10
From: SuezanneC Baskerville
Would the free Visual Studio C# 2005 Express Edition available at http://msdn.microsoft.com/vstudio/express/visualcsharp/ work alright?


It's what I use. It's prefect for the job!

:D
_____________________
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 17:26
From: Foolish Frost
It's what I use. It's prefect for the job!

:D
I'm installing it now.

If you want a real challenge, you could help me get the html code above to do something.

Free C#, Visual Basic, that Web Express thing , etc. - what a deal.

I have found from looking at source code that is available for Visual Basic that there is a lot of code available for paid versions that won't quite work on the free versions.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Travis Lambert
White dog, red collar
Join date: 3 Jun 2004
Posts: 2,819
07-08-2006 18:53
I'm installing it now as well. I'm enjoying the tutorial, Foolish! :)
_____________________
------------------
The Shelter

The Shelter is a non-profit recreation center for new residents, and supporters of new residents. Our goal is to provide a positive & supportive social environment for those looking for one in our overwhelming world.
SuezanneC Baskerville
Forums Rock!
Join date: 22 Dec 2003
Posts: 14,229
07-08-2006 19:16
From: Travis Lambert
I'm installing it now as well. I'm enjoying the tutorial, Foolish! :)

I wasn't able to just bull my way through and drop a web browser control on a form and get some action.

There are tons of tutorial videos at the MS website, I just watched the first one, which introduces one to the c# environment and gets one through a "hello world" application. The video was paced too slow for my tastes, but that pace might be alright as things get more complex.
_____________________
-

So long to these forums, the vBulletin forums that used to be at forums.secondlife.com. I will miss them.

I can be found on the web by searching for "SuezanneC Baskerville", or go to

http://www.google.com/profiles/suezanne

-

http://lindenlab.tribe.net/ created on 11/19/03.

Members: Ben, Catherine, Colin, Cory, Dan, Doug, Jim, Philip, Phoenix, Richard,
Robin, and Ryan

-
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-09-2006 05:27
From: SuezanneC Baskerville
I wasn't able to just bull my way through and drop a web browser control on a form and get some action.

There are tons of tutorial videos at the MS website, I just watched the first one, which introduces one to the c# environment and gets one through a "hello world" application. The video was paced too slow for my tastes, but that pace might be alright as things get more complex.



Never meant to indicate you would be able to from scratch, but!

I had a reason for sending you to C#. Know why? Cause a lot of the syntax is the same as LSL!

Your rampup time should be lessened by that.
_____________________
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-09-2006 05:32
I promise, as soon as I have a working basic model, I'll post it. It just takes time on my part. Time I don't have a lot of currently due to work.

Regardless, this first part is for the programmers out there. They can fill in the blanks on a lot of what I'm talking about, while new people may need more help to figure it out.

The main point is that the browser object allows a functional way to pull data from websites. That by itself is worth it since the standard HTTPWEB functions seem to be limited when dealing with multiple cookies.
_____________________
katykiwi Moonflower
Esquirette
Join date: 5 Dec 2003
Posts: 1,489
07-09-2006 15:29
Forgive my lack of geekiness, but if you can log onto the SL website under another members name and access their friends list, isnt it true that you can also access their account information? Is this a security issue that could compromise our identity, credit card info, password and provide access to our accounts being emptied?

Can you please elaborate on exactly what you are doing, in plain English, for those of us who dont speak the code you posted.
_____________________
Foolish Frost
Grand Technomancer
Join date: 7 Mar 2005
Posts: 1,433
07-09-2006 15:46
From: katykiwi Moonflower
Forgive my lack of geekiness, but if you can log onto the SL website under another members name and access their friends list, isnt it true that you can also access their account information? Is this a security issue that could compromise our identity, credit card info, password and provide access to our accounts being emptied?

Can you please elaborate on exactly what you are doing, in plain English, for those of us who dont speak the code you posted.


From: someone
"No, I'm not talking hacking, nor am I talking about invasion of anyone's privacy.

I'm talking about having software that can login to the SL website under someone's name and pull data such as his friends list."



From: someone
From: someone
Clearly in even needing to ask this I'm showing how technologically challenged I am...... but are you talking about getting a copy of your own friends list this way or someone elses?


Your own.

As to if you would be willing to share it... Well, I guess it would be possible to build a system that shared such info, and would be reasonably secure. As long as you don't (FOR GOD'S SAKE DON'T ) share your password, you should be fine.


Just to point out that was already answered in a round'bout way.

This allows the creation of software that can pull data from the website with a user name and password.

Aye, if you share your user name and password, you WILL have a problem. The fact is, this can pull the data based on that, just like the SL client can log you in with the same info.

The point is, this is a framework idea for being able to use data from the website in applications. I can't pull up your friends list/account info unless you give me your password. And you are not going to do that, right?

What about if I asked nicly? No? Ah well! :D
_____________________
Clubside Granville
Registered Bonehead
Join date: 13 Apr 2006
Posts: 478
07-09-2006 15:50
You can only log in under another user's name if you know their password, this code doesn't change that fact. It is mostly a reproduction of the login HTML necessary because most programming environments hosting a webbrowser control use ther own cookie cache rather than the one used by your regular browser.

Just a quick note for any budding developers that may not have moved to .NET: if you already own a copy of Visual Studio 6 you'll find it much easier to deal with. Whipping up an app that uses Frost's code in VB6 would take minutes. If you don't than by all means grab a .NET and see why Microsoft has always been first and foremost a tools vendor.

For Frost, you can auto-login by adding some javascript to your HTML to act upon the onLoad event of the Body tag.
_____________________
Second Life Home Page Forums - slhomepage.com

Second Life Handbook - slhandbook.com

Second Life Mainland - slmainland.com
1 2