Question

I’m taking an online class with Google’s Apex system, and would like to be able to automatically save the data from certain pages. The process of logging in and getting to the content when browsing normally is as follows: open the webapp and login, navigate the course I want to view, click the course. When I click the course I want to work on, it opens a new window with the lessons. This is the part I haven’t been able to do with a program.

The first way I thought of was with PHP, to request the web pages and simply save them. The problem is that there is a login, and a number of javascript events and things that I don’t know how to automate with php. I got as far as logging in with a POST request, but couldn’t figure the rest out.

Today I was trying to do it with Windows Forms, C#, using the dotnet WebBrowser control. I have it log in for me and navigate to the page where I need to choose the course to open, but if I click the link, it attempts to open the webpage in Internet Explorer. If I use the link it opens, I get an error from the site.

Inspecting the link on the page I’m having trouble with, I found the javascript event that opens the new window. It opens it with a redirect link. Using this redirect link in a new tab instead of new window works in Chrome, but I don’t know how to get the redirect link from C#. The a element is inside an iframe, and I have to get the link there. How can I, in C#, retrieve an element from within an iframe?

Also, is there a better way to do this?

Answer 1

Use WebClient class for getting html of url.

Example 1:

string htmlTd;

        using (WebClient client = new WebClient())
        {
         //or - request.UserAgent = "Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US)"; 
         client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
         htmlTd = client.DownloadString(myurl);
        }

       GetImagesInHTMLString(htmlTd);

// Get Images from page… it is faulty right now due to my modifications…
i am working on it, but help you get your target..

 private void GetImagesInHTMLString(string htmlString)
    {

        List<string> images = new List<string>();
        string pattern = @"<(img)\b[^>]*>";

        Regex rgx = new Regex(pattern, RegexOptions.IgnoreCase);
        MatchCollection matches = rgx.Matches(htmlString);
         string b =@"src=""";
         string c=@"src="""+myurl+"";

         //if (matches.Count >1)
         //{
            for (int i = 0, l =matches.Count; i < l; i++)
             {


                 string pattern1 =@"s/\s*src='[^']*'//";
                 //    images.Add(matches[i].Value.Replace(b, c));
                 string allmatch = matches[i].Value.Replace(b, c);
                string patrern1="#(= src=['\"].+[^\"]?)?src=[\"']?([^\"']+)#i";  
                 Regex rgx1 = new Regex(pattern1);
                 MatchCollection matches1 = rgx1.Matches(allmatch);
                 string siya = matches1[0].Value.ToString();
                 //string b = @"src=""";
                 //string c = @"src=""" + myurl + "";
             }
        // }       

        foreach (var item in images)
        {
            Response.Write(item);
        }        
    }

Example from WebClient Class link:

WebClient client = new WebClient ();

        // Add a user agent header in case the 
        // requested URI contains a query.

        client.Headers.Add ("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");

        Stream data = client.OpenRead (URl);
        StreamReader reader = new StreamReader (data);
        string s = reader.ReadToEnd ();
        Console.WriteLine (s);
        data.Close ();
        reader.Close ();