Entity Framework perf tip: Turn off AutoDetectChangesEnabled when doing bulk inserts

I was working on a situation this morning where, as part of a C# unit test method using Entity Framework 6 for database access, I was inserting about 1000 records into a SQL Server database.

This test was taking around 30-35 seconds to run, and I wanted to speed it up.  I tried re-coding the program to do the inserts using raw SQL, and that sped things up by about a factor of 3.

Thanks to a tip I found in a StackOverflow answer by “Steve”, I was able to get the same perf increase using Entity Framework by simply disabling AutoDetectChangesEnabled:

mycontext.Configuration.AutoDetectChangesEnabled = false;

Doing that before the loop with my calls to .Add() sped up the EF code to the point where it was performing just as well as the raw SQL. 

So, in situations where you’re using EF to do lots of inserts and you don’t need AutoDetectChangesEnabled (because you’re not also doing any updates to existing records), try turning it off for a possible nice performance improvement.

More info on this from Microsoft EF team member Arthur Vickers: Secrets of DetectChanges Part 3: Switching off automatic DetectChanges

Configuring ASP.NET applications in IIS to accept requests with long URLs

When creating a RESTful web service with a GET method accepts a variable-length list of parameters, it can happen that URLs generated to call the service – including the query string containing the paremters – can end up being very long. 

I’m working on a REST method with a parameter that accepts a comma-delimited list of up to 2000 ID values.  With ID values being up to 7 characters in length, the URL for a request with the maximum 2000 7-character ID values, plus an 8th comma separator character after each ID value, ends up being 16000+ charaters long.  For example:

http://mysite.example.com/api/products&productIDs=1000001,1000002,100003,…many more IDs here!…,1001999,1002000

After running into multiple obstacles trying to get my ASP.NET application running on IIS to successfully accept such incoming long request URLs without throwing errors, I’ve come to the conclusion that in situations like this, it’s better to have the API method accept HTTP POST instead of HTTP GET, and have the client pass the list of parameters in the message body intead of in the URL. This approach aligns with answers on Stackoverflow to the question Design RESTful GET API with a long list of query parameters.

However, I figured I’d go ahead and post the data I collected while troubleshooting the various errors that ASP.NET and IIS can return when a request with a long URL is submitted, in case I need this information again in the future, or in case it might help anyone else.

In all cases, the specific configured values (e.g. ”65535”) should be tailored to the specific needs of your application. Be aware that setting these values could have adverse security consequences for your application, as a large HTTP request submitted by an attacker won’t be rejected early in the pipeline as it would normally.

All of this investigation was done with an ASP.NET application targeting .NET Framework 4.5.1, running on IIS 10, on a Windows 10 64-bit PC.
 

Symptom 1

Response HTTP status code: HTTP 414

Response body: HTTP Error 414. The request URL is too long.

Relevant response header: Server: Microsoft-HTTPAPI/2.0

Fix 1

In the Windows Registry, at Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HTTP\Parameters, create a DWORD-type value with name MaxFieldLength and value sufficiently large, e.g. 65535.

Note: This error is actually thrown by http.sys, before the request even gets passed along to IIS in the request-handling pipeline.  Thus, web.config settings aren’t able to address this particluar error.  See article Check the “Server” HTTP header for the source of an error returned from IIS.

If you decide to make this change, then obviously it’ll need to be made in all environments (including all production server(s)) -- not just on your local dev PC.  Also, whatever script and/or documentation your team uses to set up new server instances will need to be updated to include this registry setting, so that your team doesn’t forget to apply this setting 18 months from now when setting up a new production server. (This was a big reason that for the API I’m building, I opted to just scrap the long-GET-URL approach, and make my method a POST instead.)

References:

• Micosoft Support article Http.sys registry settings for Windows

 

Symptom 2

Response HTTP status code: HTTP 404

Response body: (Empty)

Relevant response headers:

Server: Microsoft-IIS/10.0
X-Powered-By: ASP.NET

Fix 2

In web.config, add the following configuration (modifying existing elements when they are present, otherwise adding new elements):

<system.webserver>
  <security> 
    <requestFiltering> 
      <requestLimits maxQueryString="65535" /> 
    </requestFiltering> 
  </security> 
</system.webServer>

References:

• IIS.net documentation page on the requestLimits element
• Related Stackoverflow question: How to configure the web.config to allow requests of any length

Note: In my ASP.NET solution, I needed to make this change in my root project’s web config. This setting was ignored when I added the change in my API sub-project’s web.config.  (This however was not the case for the web.config change mentioned in “Fix 3” below.)  Related MSDN article: ASP.NET Configuration File Hierarchy and Inheritance

 

Symptom 3

Response HTTP status code: HTTP 400

Relevant response body text snippets:

System.Web.HttpException: The length of the query string for this request exceeds the configured maxQueryStringLength value.

[HttpException (0x80004005): The length of the query string for this request exceeds the configured maxQueryStringLength value.] System.Web.HttpRequest.ValidateInputIfRequiredByConfig() +492 System.Web.PipelineStepManager.ValidateHelper(HttpContext context) +55

Relevant response headers:

Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET

Fix 3

In web.config, add the following configuration (modifying existing elements when they are present, otherwise adding new elements):

<system.web> 
  <httpRuntime maxRequestLength="65535" maxUrlLength="65535" maxQueryStringLength="65535" />
</system.web>

Check the “Server” HTTP header for the source of an error returned from IIS

When an error response is returned from an HTTP request submitted to an IIS web server, the error response might actually be coming from http.sys (the “Hypertext Transfer Protocol Stack”), which processes incoming HTTP requests before they are passed along to IIS.

You can determine the source of the error by looking at the “Server” HTTP header in the returned HTTP response.  An error coming from http.sys will have the header:

Server:Microsoft-HTTPAPI/2.0

An error coming from IIS will instead have a header like:

Server:Microsoft-IIS/XX.X

Here’s some handy C# code to dump all headers from a given WebResponse to the console:

for (int i = 0; i < response.Headers.Count; i++) 
{ 
    string header = response.Headers.GetKey(i); 
    string[] values = response.Headers.GetValues(i); 
    Console.WriteLine(header + ": " + String.Join(", ", values)); 
}

Bonus tip: For a bit more information on errors generated by http.sys, check out the log files in this folder on the web server PC:

%windir%\System32\LogFiles\HTTPERR

References:

• http://stackoverflow.com/a/24179362/12484 (Thanks, snautz!)
• http://stackoverflow.com/a/32022511/12484 (Thanks, Peter Hahndorf!)
• http://stackoverflow.com/a/12284697/12484 (Thanks, ony!)