Wednesday, January 27, 2016

Entity Framework perf tip: Turn off AutoDetectChangesEnabled when doing bulk inserts

I was working on a situation this morning where, as part of a C# unit test method using Entity Framework 6 for database access, I was inserting about 1000 records into a SQL Server database.

This test was taking around 30-35 seconds to run, and I wanted to speed it up.  I tried re-coding the program to do the inserts using raw SQL, and that sped things up by about a factor of 3.

Thanks to a tip I found in a StackOverflow answer by “Steve”, I was able to get the same perf increase using Entity Framework by simply disabling AutoDetectChangesEnabled:

mycontext.Configuration.AutoDetectChangesEnabled = false;

Doing that before the loop with my calls to .Add() sped up the EF code to the point where it was performing just as well as the raw SQL. 

So, in situations where you’re using EF to do lots of inserts and you don’t need AutoDetectChangesEnabled (because you’re not also doing any updates to existing records), try turning it off for a possible nice performance improvement.

More info on this from Microsoft EF team member Arthur Vickers: Secrets of DetectChanges Part 3: Switching off automatic DetectChanges

Wednesday, January 06, 2016

Configuring ASP.NET applications in IIS to accept requests with long URLs

When creating a RESTful web service with a GET method accepts a variable-length list of parameters, it can happen that URLs generated to call the service – including the query string containing the paremters – can end up being very long. 

I’m working on a REST method with a parameter that accepts a comma-delimited list of up to 2000 ID values.  With ID values being up to 7 characters in length, the URL for a request with the maximum 2000 7-character ID values, plus an 8th comma separator character after each ID value, ends up being 16000+ charaters long.  For example:

http://mysite.example.com/api/products&productIDs=1000001,1000002,100003,…many more IDs here!…,1001999,1002000

After running into multiple obstacles trying to get my ASP.NET application running on IIS to successfully accept such incoming long request URLs without throwing errors, I’ve come to the conclusion that in situations like this, it’s better to have the API method accept HTTP POST instead of HTTP GET, and have the client pass the list of parameters in the message body intead of in the URL. This approach aligns with answers on Stackoverflow to the question Design RESTful GET API with a long list of query parameters.

However, I figured I’d go ahead and post the data I collected while troubleshooting the various errors that ASP.NET and IIS can return when a request with a long URL is submitted, in case I need this information again in the future, or in case it might help anyone else.

In all cases, the specific configured values (e.g. ”65535”) should be tailored to the specific needs of your application. Be aware that setting these values could have adverse security consequences for your application, as a large HTTP request submitted by an attacker won’t be rejected early in the pipeline as it would normally.

All of this investigation was done with an ASP.NET application targeting .NET Framework 4.5.1, running on IIS 10, on a Windows 10 64-bit PC.
 

Symptom 1

Response HTTP status code: HTTP 414

Response body: HTTP Error 414. The request URL is too long.

Relevant response header: Server: Microsoft-HTTPAPI/2.0

Fix 1

In the Windows Registry, at Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters, create a DWORD-type value with name MaxFieldLength and value sufficiently large, e.g. 65535.

Note: This error is actually thrown by http.sys, before the request even gets passed along to IIS in the request-handling pipeline.  Thus, web.config settings aren’t able to address this particluar error.  See article Check the “Server” HTTP header for the source of an error returned from IIS.

If you decide to make this change, then obviously it’ll need to be made in all environments (including all production server(s)) -- not just on your local dev PC.  Also, whatever script and/or documentation your team uses to set up new server instances will need to be updated to include this registry setting, so that your team doesn’t forget to apply this setting 18 months from now when setting up a new production server. (This was a big reason that for the API I’m building, I opted to just scrap the long-GET-URL approach, and make my method a POST instead.)

References:

 

Symptom 2

Response HTTP status code: HTTP 404

Response body: (Empty)

Relevant response headers:

Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET

Fix 2

In web.config, add the following configuration (modifying existing elements when they are present, otherwise adding new elements):

<system.webserver>
  <security> 
    <requestFiltering> 
      <requestLimits maxQueryString="65535" /> 
    </requestFiltering> 
  </security> 
</system.webServer>

References:

Note: In my ASP.NET solution, I needed to make this change in my root project’s web config. This setting was ignored when I added the change in my API sub-project’s web.config.  (This however was not the case for the web.config change mentioned in “Fix 3” below.)  Related MSDN article: ASP.NET Configuration File Hierarchy and Inheritance

 

Symptom 3

Response HTTP status code: HTTP 400

Relevant response body text snippets:

System.Web.HttpException: The length of the query string for this request exceeds the configured maxQueryStringLength value.

[HttpException (0x80004005): The length of the query string for this request exceeds the configured maxQueryStringLength value.] System.Web.HttpRequest.ValidateInputIfRequiredByConfig() +492 System.Web.PipelineStepManager.ValidateHelper(HttpContext context) +55

Relevant response headers:

Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

Fix 3

In web.config, add the following configuration (modifying existing elements when they are present, otherwise adding new elements):

<system.web> 
  <httpRuntime maxRequestLength="65535" maxUrlLength="65535" maxQueryStringLength="65535" />
</system.web>

Tuesday, January 05, 2016

Check the “Server” HTTP header for the source of an error returned from IIS

When an error response is returned from an HTTP request submitted to an IIS web server, the error response might actually be coming from http.sys (the “Hypertext Transfer Protocol Stack”), which processes incoming HTTP requests before they are passed along to IIS.

You can determine the source of the error by looking at the “Server” HTTP header in the returned HTTP response.  An error coming from http.sys will have the header:

Server:Microsoft-HTTPAPI/2.0

An error coming from IIS will instead have a header like:

Server:Microsoft-IIS/XX.X

Here’s some handy C# code to dump all headers from a given WebResponse to the console:

for (int i = 0; i < response.Headers.Count; i++) 
{ 
    string header = response.Headers.GetKey(i); 
    string[] values = response.Headers.GetValues(i); 
    Console.WriteLine(header + ": " + String.Join(", ", values)); 
}

Bonus tip: For a bit more information on errors generated by http.sys, check out the log files in this folder on the web server PC:

%windir%\System32\LogFiles\HTTPERR

References:

Wednesday, December 23, 2015

Developer Tip: Orient Your Monitor Vertically!

On my development machine, I have an external monitor physically set up in vertical / portrait orientation, instead of the “default” horizontal / landscape orientation.  My IDE (code editor) window always resides on that monitor.

Instead of writing 1000 words to explain why this works great, I’ll let a pair of pictures do most of the explaining for me.

Here’s a Visual Studio window (with the code for my open-source “Schneider’s Eleven” minimalist skin for Windows Media Player) open on my horizontal-orientation monitor at 1920x1080 resolution:

VSHorizontal2

Look at all that wasted whitespace on the right half of the main code pane!  (This code was admittedly developed using fairly short line lengths, but the same wasted space effect probably still applies to at least some extent in the majority of projects out there.)

Now, here’s the exact same Visual Studio window moved over to my vertical-orientation 1200x1920 monitor:

VSVertical1

Much less wasted whitespace!  And almost 100 lines of code are visible on the screen at the same time, making it much easier to visually scan a large chunk of source code without having to scroll around.

Having developed code this way in both Windows and Mac OS environments, I’d never go back!  Rotating a particular monitor 90 degrees is an easy configuration change in the OS display properties (in Windows, Mac OS, and evidently in Linux as well), so if your physical display stand supports it, I’d encourage giving it a try and seeing how it works for you!

Monday, December 21, 2015

Fix: “The default DbConfiguration instance was used by the Entity Framework before the '...' type was discovered” in LINQPad

For a while now, I’ve been using the excellent LINQPad to quickly run .Net Entity Framework queries using the actual object model and database mappings from one of my projects, via a DbContext connection pointed to my project’s compiled .dll file and web.config file set up in LINQPad.

Recently, after this having been working fine for a while, I started getting an error when trying to execute any queries using this DbContext connection in LINQPad:

The default DbConfiguration instance was used by the Entity Framework before the '...' type was discovered.

The "..." in that error message was actually the name of a new class that had been added to my .Net project, which derived from the DbConfiguration class.

The fix was, in my project’s web.config file, to add to the entityFramework element a codeConfigurationType attribute pointing to the new class and its package.  So, whereas before my web.config file had:

<entityFramework>
    ...
</entityFramework>

I changed it to:

<entityFramework codeConfigurationType="MyProject.MyPackage.MyCustomDBConfiguration, MyProject.MyPackage">
    ...
</entityFramework>

(Where "MyProject", "MyPackage", and "MyCustomDBConfiguration" all obviously are replaced by the actual project, package, and class names from my actual project.)

With that change in place, I was able to successfully execute queries using my project’s EF model once again.