System.NullReferenceException on Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query

[MatthewLymer]

Operating System: Ubuntu 16.04

dotnet --info

Host (useful for support):
  Version: 2.1.0
  Commit:  caa7b7e2ba

.NET Core SDKs installed:
  No SDKs were found.

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.1.0 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Dependencies:

<PackageReference Include="AutoMapper" Version="6.1.1" />
<PackageReference Include="JetBrains.Annotations" Version="11.1.0" />
<PackageReference Include="Microsoft.AspNetCore.Authentication" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Diagnostics" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Caching.Redis" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.CommandLine" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.0.0" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="2.0.0" />
<PackageReference Include="Microsoft.Net.Http.Server" Version="1.1.2" />
<PackageReference Include="Serilog.Extensions.Logging.File" Version="1.1.0" />
<PackageReference Include="FluentValidation.AspNetCore" Version="7.1.1" />
<PackageReference Include="Microsoft.Extensions.Logging.Filter" Version="1.1.2" />
<PackageReference Include="System.Text.Encoding.CodePages" Version="4.5.0" />

Exception:

System.NullReferenceException

Stack Trace:

   at Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query()
   at Microsoft.AspNetCore.Mvc.ModelBinding.QueryStringValueProviderFactory.CreateValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList``1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Cors.Infrastructure.CorsMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.ProcessRequestsAsync()

[Tratcher]

Request url? Route definitions? Action signature?

[MatthewLymer]

[Authorize]
public sealed class SearchController : Controller
{
    [HttpPost]
    [Route("Search")]
    public async Task<IActionResult> PostAsync([FromBody] SearchRequest request)
    {
        // impl
    }
}

public sealed class SearchRequest
{
    [JsonProperty(PropertyName = "Name")]
    public string Name { get; set; } 

    [JsonProperty(PropertyName = "SessionId")]
    public string SessionId { get; set; }

    [JsonProperty(PropertyName = "Tags")]
    public IDictionary<string,string> Tags { get; set; }

    [JsonProperty(PropertyName = "SearchField")]
    public string Search { get; set; }
}

The request was a POST /search HTTP/1.1 and a JSON body.

[Tratcher]

Side note: symptoms like this are most commonly caused by concurrently accessing the HttpContext on multiple threads. I don't see any other obvious signs of that in the call stack or code, but keep an eye out for it.

How consistent is the error? Every request?

[MatthewLymer]

In ~54,000 requests, it has happened ~20 times.

It looks to coincide to including authorization to my API, so I'll try and deduce from that whether it is related. I'm implementing AuthenticationHandler<T> and it's using Request.Headers and Response.Headers, not the query string specifically though.

[MatthewLymer]

I did a redeploy without authentication middleware and the issue disappeared.

[MatthewLymer]

I commented out the services.AddAuthentication(...) and app.UseAuthentication() to prevent the issue.

In my application configuration:

public void ConfigureServices(IServiceCollection services)
    {        
        services
            .AddAuthentication("Bearer")
            .AddScheme<BearerAuthenticationOptions, BearerAuthenticationHandler>(
                "Bearer", 
                settings => settings.ApiKey = Configuration["ApiKey"]);
    }
	
	
	
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory, IServiceProvider serviceProvider)
    {
        app.UseCors(b => b.AllowAnyOrigin().AllowAnyHeader().AllowAnyMethod());

        app.UseAuthentication();

        app.UseMvc();
    }
	
	
	
internal sealed class BearerAuthenticationHandler : AuthenticationHandler<BearerAuthenticationOptions>
{
    public BearerAuthenticationHandler(
        IOptionsMonitor<BearerAuthenticationOptions> options, 
        ILoggerFactory logger, 
        UrlEncoder encoder, 
        ISystemClock clock) 
        : base(options, logger, encoder, clock)
    {

    }

    protected override Task HandleChallengeAsync(AuthenticationProperties properties)
    {
        Response.Headers["WWW-Authenticate"] = "Bearer realm=\"myapp\"";

        return base.HandleChallengeAsync(properties);
    }

    protected override Task<AuthenticateResult> HandleAuthenticateAsync()
    {
        var expectedApiKey = OptionsMonitor.Get(Scheme.Name).ApiKey;
        
        if (expectedApiKey != "*")
        {
            if (!TryGetApiKey(Request, out var actualApiKey))
            {
                return Task.FromResult(AuthenticateResult.NoResult());
            }

            if (expectedApiKey != actualApiKey)
            {
                return Task.FromResult(AuthenticateResult.Fail("Bad API key."));
            }
        }

        var ticket = CreateTicket();

        return Task.FromResult(AuthenticateResult.Success(ticket));
    }

    private static bool TryGetApiKey(HttpRequest request, out string value)
    {
        const string prefix = "Bearer ";

        var authorizationValue = request.Headers["Authorization"].SingleOrDefault();

        if (authorizationValue == null)
        {
            value = null;
            return false;
        }

        if (!authorizationValue.StartsWith(prefix, StringComparison.OrdinalIgnoreCase))
        {
            value = null;
            return false;
        }

        value = authorizationValue.Substring(prefix.Length);
        return true;
    }

    private static AuthenticationTicket CreateTicket()
    {
        var identities = new[]
        {
            new ClaimsIdentity("ApiUser")
        };

        return new AuthenticationTicket(new ClaimsPrincipal(identities), "Bearer");
    }
}

[Eilon]

@MatthewLymer were you able to make any progress on this? Do you have any new information?

What @Tratcher said is pretty much always the case: some code somewhere is incorrectly holding onto an HttpContext and using it from the wrong thread or concurrently, neither of which is supported.

[Eilon]

Previous report that ended up with no firm conclusion: #2806

[allevyMS]

I am also hitting this issue, I access the HttpContext in my authorization handlers, in an action filter OnActionExecuted and in the controller method itself. I am not using the HttpContextAccessor. I assume my usage should not be concurrent?

[Eilon]

@allevyMS - can you share some relevant snippets of your code so we can take a look? Unfortunately with these kinds of issues it's really hard to diagnose without the details.

[srmorin]

We see this issue too. For us, it occurs infrequently but when it does, it always occurs in the middleware processing when trying to set the user from the claims principal in the request:

"message": "Object reference not set to an instance of an object.",
"stackTrace": "
at Microsoft.AspNetCore.Http.DefaultHttpContext.set_User(ClaimsPrincipal value)\n
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)\n
at Common.Logging.Middleware.ManageLoggingContext.Invoke(HttpContext context)\n

[Eilon]

@srmorin does any other code in your app hold onto the HttpContext (or related object) and potentially access it from another thread?

[srmorin]

Yes. We use some information from the HttpContext in our log messages, and we get this in the Invoke method in the AspDotNet core middleware.

…

________________________________ From: Eilon Lipton <notifications@github.com> Sent: Friday, July 13, 2018 4:54 PM To: aspnet/Home Cc: srmorin; Mention Subject: Re: [aspnet/Home] System.NullReferenceException on Microsoft.AspNetCore.Http.Internal.DefaultHttpRequest.get_Query (#3219) @srmorin<https://github.com/srmorin> does any other code in your app hold onto the HttpContext (or related object) and potentially access it from another thread? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#3219 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUFZmBwdP7P7hMFTjaOULdijtGZ8mS9Jks5uGQkjgaJpZM4UlLAj>.

[davidfowl]

Can you be more specific? Are you storing the http context somewhere?