Loading an url in Visual Basic 2005 Express

I am not a Visual Basic programmer, but I have downloaded and installed the free Visual Basic 2005 Express program from Microsoft with the goal of being able to read and store the text from some parts of the SL forums.

How does one read a web page in Visual Basic 20005 Express? By read, I mean get the data that would be used to display stored in Visual Basic, not displayed on the screen.

I have been doing a lot of this for my SL Friends Online App using VB Express 2005. There are a number of ways to do this depending on what sites you are trying to access and what kind of return data you need. It would be a long discussion here, but contact me in-world or offline and I'd be happy to give you more info.

Suezanne,

The key is in using the HttpWebRequest and HttpWebResponse objects to get the HTML back from a web page. Once you have it, you can then process the page. Check out this article:

http://www.vsj.co.uk/dotnet/display.asp?id=389

The main article is in C#, but also offers source code in VB.NET as well.

Here is an additional article:

http://www.csharp-station.com/HowTo/HttpWebFetch.aspx

Again, C# sorry (personally I think you are better going with C#)

Edit: here is a VB.NET article:

http://www.vbdotnetheaven.com/Code/May2004/WebClassesVBNETMCB.asp

The HttpWebRequest and HttpWebResponse objects are in the System.Net namespace.

I also have a helper class I wrote myself for another project that does what you need - the code is again C# (if you really need it in VB.NET I can rewrite it for you)- once you add this class, you can then request the contents of a web page with one line of code, like this:

string s=cWebRequest.GetPage("http://www.secondlife.com";

There is also a function if you need to post data to a page as well, PostPage.

using System;
using System.Net;
using System.IO;


namespace sluniverse
{
	public class cWebRequest
	{
		public string URL="";
		private HttpWebRequest wreq;
		
		public cWebRequest()
		{
			//
			// TODO: Add constructor logic here
			//
		}

		public cWebRequest(string url)
		{
			URL=url;


		}

		public static string PostPage(string url,string postdata)
		{
			string result="";
			HttpWebRequest r=(HttpWebRequest)WebRequest.Create(url);
			r.Method="POST";
			r.ContentLength=postdata.Length;
			r.ContentType="application/x-www-form-urlencoded";
			StreamWriter w=new StreamWriter(r.GetRequestStream());
			w.Write(postdata);
			w.Close();

			HttpWebResponse wr=(HttpWebResponse)r.GetResponse();
			using (StreamReader sr=new StreamReader(wr.GetResponseStream()) )
			{
				result=sr.ReadToEnd();
				sr.Close();

			}
			return result;





		}

		public static string GetPage(string url)
		{
			string result="";
			HttpWebRequest r=(HttpWebRequest)WebRequest.Create(url);
			r.Method="GET";
			
			HttpWebResponse wr=(HttpWebResponse)r.GetResponse();
			using (StreamReader sr=new StreamReader(wr.GetResponseStream()) )
			{
				result=sr.ReadToEnd();
				sr.Close();

			}
			return result;





		}


	}
}

This works fine if the site doesn't rely on any cookies to return the proper information. If you need to pass a cookie with the httprequest it become much more complicated.

It doesn't become much more complicated - the HttpWebRequest and HttpWebResponse objects both have a CookieContainer property that can used to retrieve and pass cookies as part of it. Here is some reference on it:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpref/html/frlrfsystemnethttpwebrequestclasscookiecontainertopic.asp

So far my tests have shown that this method only gets the new cookie data that is being passed in the webrequest. It doesn't read any exisiting cookies in the user's profiles. So in the example of the SL website, it is looking for the encoded "second-life-member" cookie to auto-log you in. THis isn't passed in the webrequest cookiecontainer, just the PHP session ID.

You have to make a call through the Winint DLL with InternetGetCookie. So far I have only seen C# and VB6 examples of this () and haven't gotten it to work in VB 2005 yet (although I have spent much of this afternoon trying).

If I am wrong please correct me Cristiano, because I would love to get this to work

The reason it is not reading in the user information from IE is because the System.Net WebRequest/Response objects are not in the context of a particular user or browser. There are a couple of things you could do in this case. You could post to the page that processes the login form, matching the post data with the name of the fields on the page and sending your user id/password, and then get back the cookie in the response to use for additional requests (the autologin). This method should work, though I have not tested it.

A second way which I have not tried but which may work is to use a tool like HttpProbe to view what the autosignin cookie value is while browsing the page with IE, and then use the CookieContainer property to send a cookie with that value as part of the request. This would then be sending the same cookie that IE sends. HttpProbe is a Java program that acts as a proxy server and lets you see the full request and response as you browse pages with IE. It is very useful for troubleshooting and debugging (I use it to check what is being passed back and forth through AJAX requests to try to optimize their size).

Another way as a last resort potentially would be to host the IE control. It has more overhead, but because of the circumstances here, the cookie requirement is a little more complicated (normally you could just set a cookie with the values you need and send it as part of the request, but since this is an autosignin cookie, you don't know the value). By using the IE control, you gain access to its cookies.

If interested, I can test out some of these methods more fully and see if I can get it to work in VB.net.

Nerds

Thanks for the response.

It is extremely apparent to me that I need to start at the beginning from VB, do some tutorials, etc.

Be that as it may, I have a project with the following code, and it displays the source for the secondlife.com site in a text box.

Imports system.net, system.io
Public Class Form1

    Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load

        Dim Req As WebRequest = WebRequest.Create("http://secondlife.com")
        Dim Resp As WebResponse
        Try
            Resp = Req.GetResponse
        Catch exc As Exception
            MsgBox(exc.Message)
            Exit Sub
        End Try
        Dim netStream As StreamReader
        netStream = New StreamReader(Resp.GetResponseStream)
        TextBox1.Text = netStream.ReadToEnd
    End Sub
End Class

However, if I try to access the forums.secondlife.com, it wants the program to log in.


(Man, there are free versions of all sorts of MS programming stuff, sheesh!)

Yes, Suezanne, that's what Cristiano and I (the nerds) were discussing above