For some strange reason I couldn’t figure out why some websites would return status 500 internal server error when they were retrieved using a WebClient in C#. The same page would render fine using a browser. It was only once in a while it happened.
I thought it might have something to do with the WebClient class so I tried using an HttpWebRequest and HttpWebResponse instead, but the result was the same. Then I started Fiddler to construct requests and tried out different HTTP headers. This let me to the problem and the solution.
The problem was that some websites use certain headers without checking if they exist or not. In this case it was the Accept-Encoding and Accept-Language headers that were missing from my request. The solution is the method below.
/// <summary>
/// Downloads a web page from the Internet and returns the HTML as a string. .
/// </summary>
/// <param name="url">The URL to download from.</param>
/// <returns>The HTML or null if the URL isn't valid.</returns>
public static string DownloadWebPage(Uri url)
{
try
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers["Accept-Encoding"] = "gzip";
request.Headers["Accept-Language"] = "en-us";
request.Credentials = CredentialCache.DefaultNetworkCredentials;
request.AutomaticDecompression = DecompressionMethods.GZip;
using (WebResponse response = request.GetResponse())
{
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
}
catch (Exception)
{
return null;
}
}
This is one of those things that seem obvious when you know the way around it. It still didn't stop me from using an hour tracking it down. Doh!
A year ago, I wrote a post about three projects that I thought would be useful for .NET developers but didn’t exist at the time. I thought it would be fun with a follow-up on that post.
1. Silverlight XHTML editor
Rich text editors like FreeTextBox and FCKeditor are extremely difficult to do right, so I wasn’t expecting a Silverlight rich text editor anytime soon. Back then Silverlight was also still in beta. A little while back I noticed that Michael Sync has started a CodePlex project of a Silverlight rich text editor. Way to go Michael!
2. OpenID membership provider for ASP.NET
I’ve really needed a plug ‘n play membership provider that supported OpenID natively. Then came along exactly that, which also is open source and available at Google Code. Good stuff, but it would be cooler if no third-party assemblies where used to make it easier to modify the code. Thumbs up anyway!
3. ASP.NET MonsterID implementation
A MonsterID is a visual image representation of an e-mail address that can be used either as a security measure or as an avatar. Only a few days after I wrote the post, Alexander Schuc wrote his MonsterID HttpHandler. This was so good that we implemented it in BlogEngine.NET 1.2. Now Gravatar supports it natively as well. Good job!
There are still a few projects that I think would be great contributions to .NET developers.
1. OAuth .NET library
The starting-to-get-popular API authentication mechanism, OAuth, would be a great addition to the .NET toolbox. Google and others are soon to support OAuth and we need both the server and client mechanisms. That would make it possible to both send and retrieve data to and from third-parties.
Update: It already exist right here code.google.com/p/devdefined-tools/wiki/OAuth
2. Semantic document writer
The relative new semantic document types such as FOAF and SIOC can be a bit tough to write yourself. They use the RDF XML format which is pretty tough to both read and write without the right tools. I would think a simple library would help democratizing these standard formats and broaden their reach. As an open source project, the library would also quickly adhere to best practices for writing these documents.
Am I missing some non-existing projects?