Developing locally with ASP.NET Core under HTTPS, SSL, and Self-Signed Certs

Last week on Twitter @getify started an excellent thread pointing out that we should be using HTTPS even on our local machines. Why? You want your local web development set up to reflect your production reality as much as possible. URL parsing, routin…

Last week on Twitter @getify started an excellent thread pointing out that we should be using HTTPS even on our local machines. Why?

You want your local web development set up to reflect your production reality as much as possible. URL parsing, routing, redirects, avoiding mixed-content warnings, etc. It's very easy to accidentally find oneself on http:// when everything in 2018 should be under https://.

I'm using ASP.NET Core 2.1 which makes local SSL super easy. After installing from http://dot.net I'll "dotnet new razor" in an empty folder to make a quick web app.

Then, when I "dotnet run" I see two URLs serving pages:

C:\Users\scott\Desktop\localsslweb> dotnet run

Hosting environment: Development
Content root path: C:\Users\scott\Desktop\localsslweb
Now listening on: https://localhost:5001
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.

One is HTTP over port 5000 and the other is HTTPS over 5001. However, if I hit https://localhost:5001, I may see an error:

Your connection to this site is not secure

That's because this is an untrusted SSL cert that was generated locally:

Untrusted cert

There's a dotnet global tool built into .NET Core 2.1 to help with certs at dev time, called "dev-certs."

C:\Users\scott> dotnet dev-certs https --help


Usage: dotnet dev-certs https [options]

Options:
-ep|--export-path Full path to the exported certificate
-p|--password Password to use when exporting the certificate with the private key into a pfx file
-c|--check Check for the existence of the certificate but do not perform any action
--clean Cleans all HTTPS development certificates from the machine.
-t|--trust Trust the certificate on the current platform
-v|--verbose Display more debug information.
-q|--quiet Display warnings and errors only.
-h|--help Show help information

I just need to run "dotnet dev-certs https --trust" and I'll get a pop up asking if I want to trust this localhost cert..

You want to trust this local cert?

On Windows it'll get added to the certificate store and on Mac it'll get added to the keychain. On Linux there isn't a standard way across distros to trust the certificate, so you'll need to perform the distro specific guidance for trusting the development certificate.

Close your browser and open up again at https://localhost:5001 and you'll see a trusted "Secure" badge in your browser.

Secure

Note also that by default HTTPS redirection is included in ASP.NET Core, and in Production it'll use HTTP Strict Transport Security (HSTS) as well, avoiding any initial insecure calls.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)

{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Error");
app.UseHsts();
}

app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseCookiePolicy();

app.UseMvc();
}

That's it. What's historically been a huge hassle for local development is essentially handled for you. Given that Chrome is marking http:// sites as "Not Secure" as of Chrome 68 you'll want to consider making ALL your sites Secure by Default. I wrote up how to get certs for free with Azure and Let's Encrypt.


Sponsor: Preview the latest JetBrains Rider with its built-in spell checking, initial Blazor support, partial C# 7.3 support, enhanced debugger, C# Interactive, and a redesigned Solution Explorer.



© 2018 Scott Hanselman. All rights reserved.
     

Example Code – Opinionated ContosoUniversity on ASP.NET Core 2.0’s Razor Pages

The best way to learn about code isn’t just writing more code – it’s reading code! Not all of it will be great code and much of it won’t be the way you would do it, but it’s a great way to expand your horizons. In fact, I’d argue that most people aren…

The best way to learn about code isn't just writing more code - it's reading code! Not all of it will be great code and much of it won't be the way you would do it, but it's a great way to expand your horizons.

In fact, I'd argue that most people aren't reading enough code. Perhaps there's not enough clean code bases to check out and learn from.

I was pleased to stumble on this code base from Jimmy Bogard called Contoso University at https://github.com/jbogard/ContosoUniversityDotNetCore-Pages.

There's a LOT of good stuff to read in this repo so I won't claim to have read it all or as deeply as I could. In fact, there's a good solid day of reading and absorbing here.However, here's some of the things I noticed and that I appreciate. Some of this is very "Jimmy" code, since it was written for and by Jimmy. This is a good thing and not a dig. We all collect patterns and make libraries and develop our own spins on architectural styles. I love that Jimmy collects a bunch of things he's created or contributed to over the years and put it into a nice clear sample for us to read. As Jimmy points out, there's a lot in https://github.com/jbogard/ContosoUniversityDotNetCore-Pages to explore:

Clone and Build just works

A low bar, right? You'd be surprised how often I git clone someone's repository and they haven't tested it elsewhere. Bonus points for a build.ps1 that bootstraps whatever needs to be done. I had .NET Core 2.x on my system already and this build.ps1 got the packages I needed and built the code cleanly.

It's an opinioned project with some opinions. ;) And that's great, because it means I'll learn about techniques and tools that I may not have used before. If someone uses a tool that's not the "defaults" it may me that the defaults are lacking!

  • Build.ps1 is using a build script style taken from PSake, a powershell build automation tool.
  • It's building to a folder called ./artifacts as as convention.
  • Inside build.ps1, it's using Roundhouse, a Database Migration Utility for .NET using sql files and versioning based on source control http://projectroundhouse.org
  • It's set up for Continuous Integration in AppVeyor, a lovely CI/CD system I use myself.
  • It uses the Octo.exe tool from OctopusDeploy to package up the artifacts.

Organized and Easy to Read

I'm finding the code easy to read for the most part. I started at Startup.cs to just get a sense of what middleware is being brought in.

public void ConfigureServices(IServiceCollection services)
{
    services.AddMiniProfiler().AddEntityFramework();
    services.AddDbContext<SchoolContext>(options =>
        options.UseSqlServer(Configuration.GetConnectionString("DefaultConnection")));
    services.AddAutoMapper(typeof(Startup));
    services.AddMediatR(typeof(Startup));
    services.AddHtmlTags(new TagConventions());
    services.AddMvc(opt =>
        {
            opt.Filters.Add(typeof(DbContextTransactionPageFilter));
            opt.Filters.Add(typeof(ValidatorPageFilter));
            opt.ModelBinderProviders.Insert(0, new EntityModelBinderProvider());
        })
        .SetCompatibilityVersion(CompatibilityVersion.Version_2_1)
        .AddFluentValidation(cfg => { cfg.RegisterValidatorsFromAssemblyContaining<Startup>(); });
}
Here I can see what libraries and helpers are being brought in, like AutoMapper, MediatR, and HtmlTags. Then I can go follow up and learn about each one.

MiniProfiler

I've always loved MiniProfiler. It's a hidden gem of .NET and it's been around being awesome forever. I blogged about it back in 2011! It sits in the corner of your web page and gives you REAL actionable details on how your site behaves and what the important perf timings are.

MiniProfiler is the profiler you didn't know you needed

It's even better with EF Core in that it'll show you the generated SQL as well! Again, all inline in your web site as you develop it.

inline SQL in MiniProfiler

Very nice.

Clean Unit Tests

Jimmy is using XUnit and has an IntegrationTestBase here with some stuff I don't understand, like SliceFixture. I'm marking this as something I need to read up on and research. I can't tell if this is the start of a new testing helper library, as it feels too generic and important to be in this sample.

He's using the CQRS "Command Query Responsibility Segregation" pattern. Here starts with a Create command, sends it, then does a Query to confirm the results. It's very clean and he's got a very isolated test.

[Fact]
public async Task Should_get_edit_details()
{
    var cmd = new Create.Command
    {
        FirstMidName = "Joe",
        LastName = "Schmoe",
        EnrollmentDate = DateTime.Today
    };
    var studentId = await SendAsync(cmd);
    var query = new Edit.Query
    {
        Id = studentId
    };
    var result = await SendAsync(query);
    result.FirstMidName.ShouldBe(cmd.FirstMidName);
    result.LastName.ShouldBe(cmd.LastName);
    result.EnrollmentDate.ShouldBe(cmd.EnrollmentDate);
}

FluentValidator

https://fluentvalidation.net is a helper library for creating clear strongly-typed validation rules. Jimmy uses it throughout and it makes for very clean validation code.

public class Validator : AbstractValidator<Command>
{
    public Validator()
    {
        RuleFor(m => m.Name).NotNull().Length(3, 50);
        RuleFor(m => m.Budget).NotNull();
        RuleFor(m => m.StartDate).NotNull();
        RuleFor(m => m.Administrator).NotNull();
    }
}

Useful Extensions

Looking at a project's C# extension methods is a great way to determine what the author feels are gaps in the underlying included functionality. These are useful for returning JSON from Razor Pages!

public static class PageModelExtensions
{
    public static ActionResult RedirectToPageJson<TPage>(this TPage controller, string pageName)
        where TPage : PageModel
    {
        return controller.JsonNet(new
            {
                redirect = controller.Url.Page(pageName)
            }
        );
    }
    public static ContentResult JsonNet(this PageModel controller, object model)
    {
        var serialized = JsonConvert.SerializeObject(model, new JsonSerializerSettings
        {
            ReferenceLoopHandling = ReferenceLoopHandling.Ignore
        });
        return new ContentResult
        {
            Content = serialized,
            ContentType = "application/json"
        };
    }
}

PaginatedList

I've always wondered what to do with helper classes like PaginatedList. Too small for a package, too specific to be built-in? What do you think?

public class PaginatedList<T> : List<T>
{
    public int PageIndex { get; private set; }
    public int TotalPages { get; private set; }
    public PaginatedList(List<T> items, int count, int pageIndex, int pageSize)
    {
        PageIndex = pageIndex;
        TotalPages = (int)Math.Ceiling(count / (double)pageSize);
        this.AddRange(items);
    }
    public bool HasPreviousPage
    {
        get
        {
            return (PageIndex > 1);
        }
    }
    public bool HasNextPage
    {
        get
        {
            return (PageIndex < TotalPages);
        }
    }
    public static async Task<PaginatedList<T>> CreateAsync(IQueryable<T> source, int pageIndex, int pageSize)
    {
        var count = await source.CountAsync();
        var items = await source.Skip((pageIndex - 1) * pageSize).Take(pageSize).ToListAsync();
        return new PaginatedList<T>(items, count, pageIndex, pageSize);
    }
}

I'm still reading all the source I can. Absorbing what resonates with me, considering what I don't know or understand and creating a queue of topics to read about. I'd encourage you to do the same! Thanks Jimmy for writing this large sample and for giving us some code to read and learn from!


Sponsor: Scale your Python for big data & big science with Intel® Distribution for Python. Near-native code speed. Use with NumPy, SciPy & scikit-learn. Get it Today!



© 2018 Scott Hanselman. All rights reserved.
     

Using Flurl to easily build URLs and make testable HttpClient calls in .NET

I posted about using Refit along with ASP.NET Core 2.1’s HttpClientFactory earlier this week. Several times when exploring this space (both on Twitter, googling around, and in my own blog comments) I come upon Flurl as in, “Fluent URL.” Not only is th…

FlurlI posted about using Refit along with ASP.NET Core 2.1's HttpClientFactory earlier this week. Several times when exploring this space (both on Twitter, googling around, and in my own blog comments) I come upon Flurl as in, "Fluent URL."

Not only is that a killer name for an open source project, Flurl is very active, very complete, and very interesting. By the way, take a look at the https://flurl.io/ site for a great example of a good home page for a well-run open source library. Clear, crisp, unambiguous, with links on how to Get It, Learn It, and Contribute. Not to mention extensive docs. Kudos!

Flurl is a modern, fluent, asynchronous, testable, portable, buzzword-laden URL builder and HTTP client library for .NET.

You had me at buzzword-laden! Flurl embraces the .NET Standard and works on .NET Framework, .NET Core, Xamarin, and UWP - so, everywhere.

To use just the Url Builder by installing Flurl. For the kitchen sink (recommended) you'll install Flurl.Http. In fact, Todd Menier was kind enough to share what a Flurl implementation of my SimpleCastClient would look like! Just to refresh you, my podcast site uses the SimpleCast podcast hosting API as its back-end.

My super basic typed implementation that "has a" HttpClient looks like this. To be clear this sample is WITHOUT FLURL.

public class SimpleCastClient

{
private HttpClient _client;
private ILogger<SimpleCastClient> _logger;
private readonly string _apiKey;

public SimpleCastClient(HttpClient client, ILogger<SimpleCastClient> logger, IConfiguration config)
{
_client = client;
_client.BaseAddress = new Uri($"https://api.simplecast.com"); //Could also be set in Startup.cs
_logger = logger;
_apiKey = config["SimpleCastAPIKey"];
}

public async Task<List<Show>> GetShows()
{
try
{
var episodesUrl = new Uri($"/v1/podcasts/shownum/episodes.json?api_key={_apiKey}", UriKind.Relative);
_logger.LogWarning($"HttpClient: Loading {episodesUrl}");
var res = await _client.GetAsync(episodesUrl);
res.EnsureSuccessStatusCode();
return await res.Content.ReadAsAsync<List<Show>>();
}
catch (HttpRequestException ex)
{
_logger.LogError($"An error occurred connecting to SimpleCast API {ex.ToString()}");
throw;
}
}
}

Let's explore Tim's expression of the same client using the Flurl library!

Not we set up a client in Startup.cs, use the same configuration, and also put in some nice aspect-oriented events for logging the befores and afters. This is VERY nice and you'll note it pulls my cluttered logging code right out of the client!

// Do this in Startup. All calls to SimpleCast will use the same HttpClient instance.

FlurlHttp.ConfigureClient(Configuration["SimpleCastServiceUri"], cli => cli
.Configure(settings =>
{
// keeps logging & error handling out of SimpleCastClient
settings.BeforeCall = call => logger.LogWarning($"Calling {call.Request.RequestUri}");
settings.OnError = call => logger.LogError($"Call to SimpleCast failed: {call.Exception}");
})
// adds default headers to send with every call
.WithHeaders(new
{
Accept = "application/json",
User_Agent = "MyCustomUserAgent" // Flurl will convert that underscore to a hyphen
}));

Again, this set up code lives in Startup.cs and is a one-time thing. The Headers, User Agent all are dealt with once there and in a one-line chained "fluent" manner.

Here's the new SimpleCastClient with Flurl.

using Flurl;

using Flurl.Http;

public class SimpleCastClient
{
// look ma, no client!
private readonly string _baseUrl;
private readonly string _apiKey;

public SimpleCastClient(IConfiguration config)
{
_baseUrl = config["SimpleCastServiceUri"];
_apiKey = config["SimpleCastAPIKey"];
}

public Task<List<Show>> GetShows()
{
return _baseUrl
.AppendPathSegment("v1/podcasts/shownum/episodes.json")
.SetQueryParam("api_key", _apiKey)
.GetJsonAsync<List<Show>>();
}
}

See in GetShows() how we're also using the Url Builder fluent extensions in the Flurl library. See that _baseUrl is actually a string? We all know that we're supposed to use System.Uri but it's such a hassle. Flurl adds extension methods to strings so that you can seamlessly transition from the strings (that we all use) representations of Urls/Uris and build up a Query String, and in this case, a GET that returns JSON.

Very clean!

Flurl also prides itself on making HttpClient testing easier as well. Here's a more sophisticated example of a library from their site:

// Flurl will use 1 HttpClient instance per host

var person = await "https://api.com"
.AppendPathSegment("person")
.SetQueryParams(new { a = 1, b = 2 })
.WithOAuthBearerToken("my_oauth_token")
.PostJsonAsync(new
{
first_name = "Claire",
last_name = "Underwood"
})
.ReceiveJson<Person>();

This example is doing a post with an anonymous object that will automatically turn into JSON when it hits the wire. It also receives JSON as the response. Even the query params are created with a C# POCO (Plain Old CLR Object) and turned into name=value strings automatically.

Here's a test Flurl-style!

// fake & record all http calls in the test subject

using (var httpTest = new HttpTest()) {
// arrange
httpTest.RespondWith(200, "OK");
// act
await sut.CreatePersonAsync();
// assert
httpTest.ShouldHaveCalled("https://api.com/*")
.WithVerb(HttpMethod.Post)
.WithContentType("application/json");
}

Flurl.Http includes a set of features to easily fake and record HTTP activity. You can make a whole series of assertions about your APIs:

httpTest.ShouldHaveCalled("http://some-api.com/*")

.WithVerb(HttpMethd.Post)
.WithContentType("application/json")
.WithRequestBody("{\"a\":*,\"b\":*}") // supports wildcards
.Times(1)

All in all, it's an impressive set of tools that I hope you explore and consider for your toolbox! There's a ton of great open source like this with .NET Core and I'm thrilled to do a small part to spread the word. You should to!


Sponsor: Check out dotMemory Unit, a free unit testing framework for fighting all kinds of memory issues in your code. Extend your unit testing with the functionality of a memory profiler.



© 2018 Scott Hanselman. All rights reserved.
     

Using ASP.NET Core 2.1’s HttpClientFactory with Refit’s REST library

When I moved my podcast site over to ASP.NET Core 2.1 I also started using HttpClientFactory and wrote up my experience. It’s a nice clean way to centralize both settings and policy for your HttpClients, especially if you’re using a lot of them to talk…

Strong by Lucyb_22 used under Creative Commons from FlickrWhen I moved my podcast site over to ASP.NET Core 2.1 I also started using HttpClientFactory and wrote up my experience. It's a nice clean way to centralize both settings and policy for your HttpClients, especially if you're using a lot of them to talk to a lot of small services.

Last year I explored Refit, an automatic type-safe REST library for .NET Standard. It makes it super easy to just declare the shape of a client and its associated REST API with a C# interface:

public interface IGitHubApi

{
[Get("/users/{user}")]
Task<User> GetUser(string user);
}

and then ask for an HttpClient that speaks that API's shape, then call it. Fabulous.

var gitHubApi = RestService.For<IGitHubApi>("https://api.github.com");


var octocat = await gitHubApi.GetUser("octocat");

But! What does Refit look like and how does it work in an HttpClientFactory-enabled world? Refit has recently been updated with first class support for ASP.NET Core 2.1's HttpClientFactory with the Refit.HttpClientFactory package.

Since you'll want to centralize all your HttpClient configuration in your ConfigureServices method in Startup, Refit adds a nice extension method hanging off of Services.

You add a RefitClient of a type, then add whatever other IHttpClientBuilder methods you want afterwards:

services.AddRefitClient<IWebApi>()

.ConfigureHttpClient(c => c.BaseAddress = new Uri("https://api.example.com"));
// Add additional IHttpClientBuilder chained methods as required here:
// .AddHttpMessageHandler<MyHandler>()
// .SetHandlerLifetime(TimeSpan.FromMinutes(2));

Of course, then you can just have your HttpClient automatically created and passed into the constructor. You'll see in this sample from their GitHub that you get an IWebAPI (that is, whatever type you want, like my IGitHubApi) and just go to town with a strongly typed interfaces of an HttpClient with autocomplete.

public class HomeController : Controller

{
public HomeController(IWebApi webApi)
{
_webApi = webApi;
}

private readonly IWebApi _webApi;

public async Task<IActionResult> Index(CancellationToken cancellationToken)
{
var thing = await _webApi.GetSomethingWeNeed(cancellationToken);

return View(thing);
}
}

Refit is easy to use, and even better with ASP.NET Core 2.1. Go get Refit and try it today!

* Strong image by Lucyb_22 used under Creative Commons from Flickr


Sponsor: Check out dotMemory Unit, a free unit testing framework for fighting all kinds of memory issues in your code. Extend your unit testing with the functionality of a memory profiler.



© 2018 Scott Hanselman. All rights reserved.
     

ASP.NET Core Architect David Fowler’s hidden gems in 2.1

Open source ASP.NET Core 2.1 is out, and Architect David Fowler took to twitter to share some hidden gems that not everyone knows about. Sure, it’s faster, builds faster, runs faster, but there’s a number of details and fun advanced techniques that are…

ASP.NET Architect David FowlerOpen source ASP.NET Core 2.1 is out, and Architect David Fowler took to twitter to share some hidden gems that not everyone knows about. Sure, it's faster, builds faster, runs faster, but there's a number of details and fun advanced techniques that are worth a closer look at.

.NET Generic Host

ASP.NET Core introduced a new hosting model. .NET apps configure and launch a host.

The host is responsible for app startup and lifetime management. The goal of the Generic Host is to decouple the HTTP pipeline from the Web Host API to enable a wider array of host scenarios. Messaging, background tasks, and other non-HTTP workloads based on the Generic Host benefit from cross-cutting capabilities, such as configuration, dependency injection (DI), and logging.

This means that there's not just a WebHost anymore, there's a Generic Host for non-web-hosting scenarios. You get the same feeling as with ASP.NET Core and all the cool features like DI, logging, and config. The sample code for a Generic Host is up on GitHub.

IHostedService

A way to run long running background operations in both the generic host and in your web hosted applications. ASP.NET Core 2.1 added support for a BackgroundService base class that makes it trivial to write a long running async loop. The sample code for a Hosted Service is also up on GitHub.

Check out a simple Timed Background Task:

public Task StartAsync(CancellationToken cancellationToken)

{
_logger.LogInformation("Timed Background Service is starting.");

_timer = new Timer(DoWork, null, TimeSpan.Zero,
TimeSpan.FromSeconds(5));

return Task.CompletedTask;
}

Fun!

Windows Services on .NET Core

You can now host ASP.NET Core inside a Windows Service! Lots of people have been asking for this. Again, no need for IIS, and you can host whatever makes you happy. Check out Microsoft.AspNetCore.Hosting.WindowsServices on NuGet and extensive docs on how to host your own ASP.NET Core app without IIS on Windows as a Windows Service.

public static void Main(string[] args)

{
var pathToExe = Process.GetCurrentProcess().MainModule.FileName;
var pathToContentRoot = Path.GetDirectoryName(pathToExe);

var host = WebHost.CreateDefaultBuilder(args)
.UseContentRoot(pathToContentRoot)
.UseStartup<Startup>()
.Build();

host.RunAsService();
}

IHostingStartup - Configure IWebHostBuilder with an Assembly Attribute

Simple and clean with source on GitHub as always.

[assembly: HostingStartup(typeof(SampleStartups.StartupInjection))]

Shared Source Packages

This is an interesting one you should definitely take a moment and pay attention to. It's possible to build packages that are used as helpers to share source code. We internally call these "shared source packages." These are used all over ASP.NET Core for things that should be shared BUT shouldn't be public APIs. These get used but won't end up as actual dependencies of your resulting package.

They are consumed like this in a CSPROJ. Notice the PrivateAssets attribute.

<PackageReference Include="Microsoft.Extensions.ClosedGenericMatcher.Sources" PrivateAssets="All" Version="" />

<PackageReference Include="Microsoft.Extensions.ObjectMethodExecutor.Sources" PrivateAssets="All" Version="" />

ObjectMethodExecutor

If you ever need to invoke a method on a type via reflection and that method could be async, we have a helper that we use everywhere in the ASP.NET Core code base that is highly optimized and flexible called the ObjectMethodExecutor.

The team uses this code in MVC to invoke your controller methods. They use this code in SignalR to invoke your hub methods. It handles async and sync methods. It also handles custom awaitables and F# async workflows

SuppressStatusMessages

A small and commonly requested one. If you hate the output that dotnet run gives when you host a web application (printing out the binding information) you can use the new SuppressStatusMessages extension method.

WebHost.CreateDefaultBuilder(args)

.SuppressStatusMessages(true)
.UseStartup<Startup>();

AddOptions

They made it easier in 2.1 to configure options that require services. Previously, you would have had to create a type that derived from IConfigureOptions<TOptions>, now you can do it all in ConfigureServices via AddOptions<TOptions>

public void ConfigureServicdes(IServiceCollection services)

{
services.AddOptions<MyOptions>()
.Configure<IHostingEnvironment>((o,env) =>
{
o.Path = env.WebRootPath;
});
}

IHttpContext via AddHttpContextAccessor

You likely shouldn't be digging around for IHttpContext, but lots of folks ask how to get to it and some feel it should be automatic. It's not registered by default since having it has a performance cost. However, in ASP.NET Core 2.1 a PR was put in for an extension method that makes it easy IF you want it.

services.AddHttpContextAccessor();

So ASP.NET Core 2.1 is out and ready to go

New features in this release include:

Check out What's New in ASP.NET Core 2.1 in the ASP.NET Core docs to learn more about these features. For a complete list of all the changes in this release, see the release notes.

Go give it a try. Follow this QuickStart and you can have a basic Web App up in 10 minutes.


Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!



© 2018 Scott Hanselman. All rights reserved.
     

Real Browser Integration Testing with Selenium Standalone, Chrome, and ASP.NET Core 2.1

Buckle up kids, this is nuts and I’m probably doing it wrong. 😉 And it’s 2am and I wrote this fast. I’ll come back tomorrow and fix the spelling. I want to have lots of tests to make sure my new podcast site is working well. As mentioned before, I’ve …

I find your lack of tests disturbingBuckle up kids, this is nuts and I'm probably doing it wrong. ;) And it's 2am and I wrote this fast. I'll come back tomorrow and fix the spelling.

I want to have lots of tests to make sure my new podcast site is working well. As mentioned before, I've been updating the site to ASP.NET Core 2.1.

Here's some posts if you want to catch up:

I've been doing my testing with XUnit and I want to test in layers.

Basic Unit Testing

Simply create a Razor Page's Model in memory and call OnGet or WhateverMethod. At this point you are NOT calling Http, there is no WebServer.

public IndexModel pageModel;


public IndexPageTests()
{
var testShowDb = new TestShowDatabase();
pageModel = new IndexModel(testShowDb);
}

[Fact]
public async void MainPageTest()
{
// FAKE HTTP GET "/"
IActionResult result = await pageModel.OnGetAsync(null, null);

Assert.NotNull(result);
Assert.True(pageModel.OnHomePage); //we are on the home page, because "/"
Assert.Equal(16, pageModel.Shows.Count()); //home page has 16 shows showing
Assert.Equal(620, pageModel.LastShow.ShowNumber); //last test show is #620
}

Moving out a layer...

In-Memory Testing with both Client and Server using WebApplicationFactory

Here we are starting up the app and calling it with a client, but the "HTTP" of it all is happening in memory/in process. There are no open ports, there's no localhost:5000. We can still test HTTP semantics though.

public class TestingFunctionalTests : IClassFixture<WebApplicationFactory<Startup>>

{
public HttpClient Client { get; }
public ServerFactory<Startup> Server { get; }

public TestingFunctionalTests(ServerFactory<Startup> server)
{
Client = server.CreateClient();
Server = server;
}

[Fact]
public async Task GetHomePage()
{
// Arrange & Act
var response = await Client.GetAsync("/");

// Assert
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
...
}

Testing with a real Browser and real HTTP using Selenium Standalone and Chrome

THIS is where it gets interesting with ASP.NET Core 2.1 as we are going to fire up both the complete web app, talking to the real back end (although it could talk to a local test DB if you want) as well as a real headless version of Chrome being managed by Selenium Standalone and talked to with the WebDriver. It sounds complex, but it's actually awesome and super useful.

First I add references to Selenium.Support and Selenium.WebDriver to my Test project:

dotnet add reference "Selenium.Support"

dotnet add reference "Selenium.WebDriver"

Make sure you have node and npm then you can get Selenium Standalone like this:

npm install -g [email protected]

selenium-standalone install

Chrome is being controlled by automated test softwareSelenium, to be clear, puts your browser on a puppet's strings. Even Chrome knows it's being controlled! It's using the (soon to be standard, but clearly defacto standard) WebDriver protocol. Imagine if your browser had a localhost REST protocol where you could interrogate it and click stuff! I've been using Selenium for over 11 years. You can even test actual Windows apps (not in the browser) with WinAppDriver/Appium but that's for another post.

Now for this part, bare with me because my ServerFactory class I'm about to make is doing two things. It's setting up my ASP.NET Core 2. 1 app and actually running it so it's listening on https://localhost:5001. It's assuming a few things that I'll point out. It also (perhaps questionable) is launching Selenium Standalone from within its constructor. Questionable, to be clear, and there's others ways to do this, but this is VERY simple.

If it offends you, remembering that you do need to start Selenium Standalone with "selenium-standalone start" you could do it OUTSIDE your test in a script.

Perhaps do the startup/teardown work in a PowerShell or Shell script. Start it up, save the process id, then stop it when you're done. Note I'm also doing checking code coverage here with Coverlet but that's not related to Selenium - I could just "dotnet test."

#!/usr/local/bin/powershell

$SeleniumProcess = Start-Process "selenium-standalone" -ArgumentList "start" -PassThru
dotnet test /p:CollectCoverage=true /p:CoverletOutputFormat=lcov /p:CoverletOutput=./lcov .\hanselminutes.core.tests
Stop-Process -Id $SeleniumProcess.Id

Here my SeleniumServerFactory is getting my Browser and Server ready.

SIDEBAR NOTE: I want to point out that this is NOT perfect and it's literally the simplest thing possible to get things working. It's my belief, though, that there are some problems here and that I shouldn't have to fake out the "new TestServer" in CreateServer there. While the new WebApplicationFactory is great for in-memory unit testing, it should be just as easy to fire up your app and use a real port for things like Selenium testing. Here I'm building and starting the IWebHostBuilder myself (!) and then making a fake TestServer only to satisfy the CreateServer method, which I think should not have a concrete class return type. For testing, ideally I could easily get either an "InMemoryWebApplicationFactory" and a "PortUsingWebApplicationFactory" (naming is hard). Hopefully this is somewhat clear and something that can be easily adjusted for ASP.NET Core 2.1.x.

My app is configured to listen on both http://localhost:5000 and https://localhost:5001, so you'll note where I'm getting that last value (in an attempt to avoid hard-coding it). We also are sure to stop both Server and Brower in Dispose() at the bottom.

public class SeleniumServerFactory<TStartup> : WebApplicationFactory<Startup> where TStartup : class

{
public string RootUri { get; set; } //Save this use by tests

Process _process;
IWebHost _host;

public SeleniumServerFactory()
{
ClientOptions.BaseAddress = new Uri("https://localhost"); //will follow redirects by default

_process = new Process() {
StartInfo = new ProcessStartInfo {
FileName = "selenium-standalone",
Arguments = "start",
UseShellExecute = true
}
};
_process.Start();
}

protected override TestServer CreateServer(IWebHostBuilder builder)
{
//Real TCP port
_host = builder.Build();
_host.Start();
RootUri = _host.ServerFeatures.Get<IServerAddressesFeature>().Addresses.LastOrDefault(); //Last is https://localhost:5001!

//Fake Server we won't use...this is lame. Should be cleaner, or a utility class
return new TestServer(new WebHostBuilder().UseStartup<TStartup>());
}

protected override void Dispose(bool disposing)
{
        base.Dispose(disposing);
        if (disposing) {
            _host.Dispose();
_process.CloseMainWindow(); //Be sure to stop Selenium Standalone
        }
    }
}

But what does a complete series of tests look like? I have a Server, a Browser, and an (theoretically optional) HttpClient. Focus on the Browser and Server.

At the point when a single test starts, my site is up (the Server) and an invisible headless Chrome (the Browser) is actually being puppeted with local calls via WebDriver. All this is hidden from to you - if you want. You can certainly see Chrome (or other browsers) get automated, but what's nice about Selenium Standalone with hidden/headless Browser testing is that my unit tests now also include these complete Integration Tests and can run as part of my Continuous Integration Build.

Again, layers. I test classes, then move out and test Http Request/Response interactions, and finally the site is up and I'm making sure I can navigate, that data is loading. I'm automating the "smoke tests" that I used to do myself! And I can make as many of this a I'd like now that the scaffolding work is done.

public class SeleniumTests : IClassFixture<SeleniumServerFactory<Startup>>, IDisposable

{
public SeleniumServerFactory<Startup> Server { get; }
public IWebDriver Browser { get; }
public HttpClient Client { get; }
public ILogs Logs { get; }

public SeleniumTests(SeleniumServerFactory<Startup> server)
{
Server = server;
Client = server.CreateClient(); //weird side effecty thing here. This call shouldn't be required for setup, but it is.

var opts = new ChromeOptions();
opts.AddArgument("--headless"); //Optional, comment this out if you want to SEE the browser window
opts.SetLoggingPreference(OpenQA.Selenium.LogType.Browser, LogLevel.All);

var driver = new RemoteWebDriver(opts);
Browser = driver;
Logs = new RemoteLogs(driver); //TODO: Still not bringing the logs over yet
}

[Fact]
public void LoadTheMainPageAndCheckTitle()
{
Browser.Navigate().GoToUrl(Server.RootUri);
Assert.StartsWith("Hanselminutes Technology Podcast - Fresh Air and Fresh Perspectives for Developers", Browser.Title);
}

[Fact]
public void ThereIsAnH1()
{
Browser.Navigate().GoToUrl(Server.RootUri);

var headerSelector = By.TagName("h1");
Assert.Equal("HANSELMINUTES PODCAST\r\nby Scott Hanselman", Browser.FindElement(headerSelector).Text);
}

[Fact]
public void KevinScottTestThenGoHome()
{
Browser.Navigate().GoToUrl(Server.RootUri + "/631/how-do-you-become-a-cto-with-microsofts-cto-kevin-scott");

var headerSelector = By.TagName("h1");
var link = Browser.FindElement(headerSelector);
link.Click();
Assert.Equal(Browser.Url.TrimEnd('/'),Server.RootUri); //WTF
}

public void Dispose()
{
Browser.Dispose();
}
}

Here's a build, unit test/selenium test with code coverage actually running. I started running it from PowerShell. The black window in the back is Selenium Standalone doing its thing (again, could be hidden).

Two consoles, one with PowerShell running XUnit and one running Selenium

If I comment out the "--headless" line, I'll see this as Chrome is automated. Cool.

Chrome is loading my site and being automated

Of course, I can also run these in the .NET Core Test Explorer in either Visual Studio Code, or Visual Studio.

image

Great fun. What are your thoughts?


Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!



© 2018 Scott Hanselman. All rights reserved.
     

Using LazyCache for clean and simple .NET Core in-memory caching

I’m continuing to use .NET Core 2.1 to power my Podcast Site, and I’ve done a series of posts on some of the experiments I’ve been doing. I also upgraded to .NET Core 2.1 RC that came out this week. Here’s some posts if you want to catch up: Eyes wide…

Tai Chi by Luisen Rodrigo - Used under CCI'm continuing to use .NET Core 2.1 to power my Podcast Site, and I've done a series of posts on some of the experiments I've been doing. I also upgraded to .NET Core 2.1 RC that came out this week. Here's some posts if you want to catch up:

Having a blast, if I may say so.

I've been trying a number of ways to cache locally. I have an expensive call to a backend (7-8 seconds or more, without deserialization) so I want to cache it locally for a few hours until it expires. I have a way that work very well using a SemaphoreSlim. There's some issues to be aware of but it has been rock solid. However, in the comments of the last caching post a number of people suggested I use "LazyCache."

Alastair from the LazyCache team said this in the comments:

LazyCache wraps your "build stuff I want to cache" func in a Lazy<> or an AsyncLazy<> before passing it into MemoryCache to ensure the delegate only gets executed once as you retrieve it from the cache. It also allows you to swap between sync and async for the same cached thing. It is just a very thin wrapper around MemoryCache to save you the hassle of doing the locking yourself. A netstandard 2 version is in pre-release.
Since you asked the implementation is in CachingService.cs#L119 and proof it works is in CachingServiceTests.cs#L343

Nice! Sounds like it's worth trying out. Most importantly, it'll allow me to "refactor via subtraction."

I want to have my "GetShows()" method go off and call the backend "database" which is a REST API over HTTP living at SimpleCast.com. That backend call is expensive and doesn't change often. I publish new shows every Thursday, so ideally SimpleCast would have a standard WebHook and I'd cache the result forever until they called me back. For now I will just cache it for 8 hours - a long but mostly arbitrary number. Really want that WebHook as that's the correct model, IMHO.

LazyCache was added on my Configure in Startup.cs:

services.AddLazyCache();

Kind of anticlimactic. ;)

Then I just make a method that knows how to populate my cache. That's just a "Func" that returns a Task of List of Shows as you can see below. Then I call IAppCache's "GetOrAddAsync" from LazyCache that either GETS the List of Shows out of the Cache OR it calls my Func, does the actual work, then returns the results. The results are cached for 8 hours. Compare this to my previous code and it's a lot cleaner.

public class ShowDatabase : IShowDatabase
{
    private readonly IAppCache _cache;
    private readonly ILogger _logger;
    private SimpleCastClient _client;
    public ShowDatabase(IAppCache appCache,
            ILogger<ShowDatabase> logger,
            SimpleCastClient client)
    {
        _client = client;
        _logger = logger;
        _cache = appCache;
    }
    public async Task<List<Show>> GetShows()
    {    
        Func<Task<List<Show>>> showObjectFactory = () => PopulateShowsCache();
        var retVal = await _cache.GetOrAddAsync("shows", showObjectFactory, DateTimeOffset.Now.AddHours(8));
        return retVal;
    }
 
    private async Task<List<Show>> PopulateShowsCache()
    {
        List<Show> shows = shows = await _client.GetShows();
        _logger.LogInformation($"Loaded {shows.Count} shows");
        return shows.Where(c => c.PublishedAt < DateTime.UtcNow).ToList();
    }
}

It's always important to point out there's a dozen or more ways to do this. I'm not selling a prescription here or The One True Way, but rather exploring the options and edges and examining the trade-offs.

  • As mentioned before, me using "shows" as a magic string for the key here makes no guarantees that another co-worker isn't also using "shows" as the key.
    • Solution? Depends. I could have a function-specific unique key but that only ensures this function is fast twice. If someone else is calling the backend themselves I'm losing the benefits of a centralized (albeit process-local - not distributed like Redis) cache.
  • I'm also caching the full list and then doing a where/filter every time.
    • A little sloppiness on my part, but also because I'm still feeling this area out. Do I want to cache the whole thing and then let the callers filter? Or do I want to have GetShows() and GetActiveShows()? Dunno yet. But worth pointing out.
  • There's layers to caching. Do I cache the HttpResponse but not the deserialization? Here I'm caching the List<Shows>, complete. I like caching List<T> because a caller can query it, although I'm sending back just active shows (see above).
    • Another perspective is to use the <cache> TagHelper in Razor and cache Razor's resulting rendered HTML. There is value in caching the object graph, but I need to think about perhaps caching both List<T> AND the rendered HTML.
    • I'll explore this next.

I'm enjoying myself though. ;)

Go explore LazyCache! I'm using beta2 but there's a whole number of releases going back years and it's quite stable so far.

Lazy cache is a simple in-memory caching service. It has a developer friendly generics based API, and provides a thread safe cache implementation that guarantees to only execute your cachable delegates once (it's lazy!). Under the hood it leverages ObjectCache and Lazy to provide performance and reliability in heavy load scenarios.

For ASP.NET Core it's quick to experiment with LazyCache and get it set up. Give it a try, and share your favorite caching techniques in the comments.

Tai Chi photo by Luisen Rodrigo used under Creative Commons Attribution 2.0 Generic (CC BY 2.0), thanks!


Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!



© 2018 Scott Hanselman. All rights reserved.
     

Announcing .NET Core 2.1 RC 1 Go Live AND .NET Core 3.0 Futures

I just got back from the Microsoft BUILD Conference where Scott Hunter and I announced both .NET Core 2.1 RC1 AND talked about .NET Core 3.0 in the future. .NET Core 2.1 RC1 First, .NET Core 2.1’s Release Candidate is out. This one has a Go Live licen…

I just got back from the Microsoft BUILD Conference where Scott Hunter and I announced both .NET Core 2.1 RC1 AND talked about .NET Core 3.0 in the future.

.NET Core 2.1 RC1

First, .NET Core 2.1's Release Candidate is out. This one has a Go Live license and it's very close to release.

You can download and get started with .NET Core 2.1 RC 1, on Windows, macOS, and Linux:

You can see complete details of the release in the .NET Core 2.1 RC 1 release notes. Related instructions, known issues, and workarounds are included in releases notes. Please report any issues you find in the comments or at dotnet/core #1506. ASP.NET Core 2.1 RC 1 and Entity Framework 2.1 RC 1 are also releasing today. You can develop .NET Core 2.1 apps with Visual Studio 2017 15.7, Visual Studio for Mac 7.5, or Visual Studio Code.

Here's a deep dive on the performance benefits which are SIGNIFICANT. It's also worth noting that you can get 2x+ speed improvements for your builds/compiles, but and to adopt the .NET Core 2.1 RC SDK while continuing to target earlier .NET Core releases, like 2.0 for the Runtime.

  • Go Live - You can put this version in production and get support.
  • Alpine Support - There are docker images at 2.1-sdk-alpine and 2.1-runtime-alpine.
  • ARM Support - We can compile on Raspberry Pi now! .NET Core 2.1 is supported on Raspberry Pi 2+. It isn’t supported on the Pi Zero or other devices that use an ARMv6 chip. .NET Core requires ARMv7 or ARMv8 chips, like the ARM Cortex-A53. There are even Docker images for ARM32
  • Brotli Support - new lossless compression algo for the web.
  • Tons of new Crypto Support.
  • Source Debugging from NuGet Packages (finally!) called "SourceLink."
  • .NET Core Global Tools:
  • dotnet tool install -g dotnetsay
    dotnetsay

In fact, if you have Docker installed go try an ASP.NET Sample:

docker pull microsoft/dotnet-samples:aspnetapp
docker run --rm -it -p 8000:80 --name aspnetcore_sample microsoft/dotnet-samples:aspnetapp

.NET Core 3.0

This is huge. You'll soon be able to take your existing WinForms and WPF app (I did this with a 12 year old WPF app!) and swap out the underlying runtime. That means you can run WinForms and WPF on .NET Core 3 on Windows.

Why is this cool?

  • WinForms/WPF apps can but self-contained and run in a single folder.

No need to install anything, just xcopy deploy. WinFormsApp1 can't affect WPFApp2 because they can each target their own .NET Core 3 version. Updates to the .NET Framework on Windows are system-wide and can sometimes cause problems with legacy apps. You'll now have total control and update apps one at at time and they can't affect each other. C#, F# and VB already work with .NET Core 2.0. You will be able to build desktop applications with any of those three languages with .NET Core 3.

Secondly, you'll get to use all the new C# 7.x+ (and beyond) features sooner than ever. .NET Core moves fast but you can pick and choose the language features and libraries you want. For example, I can update BabySmash (my .NET 3.5 WPF app) to .NET Core 3.0 and use new C# features AND bring in UWP Controls that didn't exist when BabySmash was first written! WinForms and WPF apps will also get the new lightweight csproj format. More details here and a full video below.

  • Compile to a single EXE

Even more, why not compile the whole app into a single EXE. I can make BabySmash.exe and it'll just work. No install, everything self-contained.

.NET Core 3 will still be cross platform, but WinForms and WPF remain "W is for Windows" - the runtime is swappable, but they still P/Invoke into the Windows APIs. You can look elsewhere for .NET Core cross-platform UI apps with frameworks like Avalonia, Ooui, and Blazor.

Diagram showing that .NET Core will support Windows UI Frameworks

You can check out the video from BUILD here. We show 2.1, 3.0, and some amazing demos like compiling a .NET app into a single exe and running it on a computer from the audience, as well as taking the 12 year old BabySmash WPF app and running it on .NET Core 3.0 PLUS adding a UWP Touch Ink Control!

Lots of cool stuff coming today AND tomorrow with open source .NET Core!


Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!



© 2018 Scott Hanselman. All rights reserved.
     

The Programmer’s Hindsight – Caching with HttpClientFactory and Polly Part 2

In my last blog post Adding Cross-Cutting Memory Caching to an HttpClientFactory in ASP.NET Core with Polly I actually failed to complete my mission. I talked to a few people (thanks Dylan and Damian and friends) and I think my initial goal may have be…

Hindsight - by Nate Steiner - Public DomainIn my last blog post Adding Cross-Cutting Memory Caching to an HttpClientFactory in ASP.NET Core with Polly I actually failed to complete my mission. I talked to a few people (thanks Dylan and Damian and friends) and I think my initial goal may have been wrong.

I thought I wanted "magic add this policy and get free caching" for HttpClients that come out of the new .NET Core 2.1 HttpClientFactory, but first, nothing is free, and second, everything (in computer science) is layered. Am I caching the right thing at the right layer?

The good thing that come out of explorations and discussions like this is Better Software. Given that I'm running Previews/Daily Builds of both .NET Core 2.1 (in preview as of the time of this writing) and Polly (always under active development) I realize I'm on some kind of cutting edge. The bad news (and it's not really bad) is that everything I want to do is possible it's just not always easy. For example, a lot of "hooking up" happens when one make a C# Extension Method and adds in into the ASP.NET Middleware Pipeline with "services.AddSomeStuffThatIsTediousButUseful()."

Polly and ASP.NET Core are insanely configurable, but I'm personally interested in the 80% or even the 90% case. The 10% will definitely require you/me to learn more about the internals of the system, while the 90% will ideally be abstracted away from the average developer (me).

I've had a Skype with Dylan from Polly and he's been updating the excellent Polly docs as we walk around how caching should work in an HttpClientFactory world. Fantastic stuff, go read it. I'll steal some here:

ASPNET Core 2.1 - What is HttpClient factory?

From ASPNET Core 2.1, Polly integrates with IHttpClientFactory. HttpClient factory is a factory that simplifies the management and usage of HttpClient in four ways. It:

  • allows you to name and configure logical HttpClients. For instance, you may configure a client that is pre-configured to access the github API;

  • manages the lifetime of HttpClientMessageHandlers to avoid some of the pitfalls associated with managing HttpClient yourself (the dont-dispose-it-too-often but also dont-use-only-a-singleton aspects);

  • provides configurable logging (via ILogger) for all requests and responses performed by clients created with the factory;

  • provides a simple API for adding middleware to outgoing calls, be that for logging, authorisation, service discovery, or resilience with Polly.

The Microsoft early announcement speaks more to these topics, and Steve Gordon's pair of blog posts (1; 2) are also an excellent read for deeper background and some great worked examples.

Polly and Polly policies work great with ASP.NET Core 2.1 and integrated nicely. I'm sure it will integrate even more conveniently with a few smart Extension Methods to abstract away the hard parts so we can fall into the "pit of success."

Caching with Polly and HttpClient

Here's where it gets interesting. To me. Or, you, I suppose, Dear Reader, if you made it this far into a blog post (and sentence) with too many commas.

This is a salient and important point:

Polly is generic (not tied to Http requests)

Now, this is where I got in trouble:

Caching with Polly CachePolicy in a DelegatingHandler caches at the HttpResponseMessage level

I ended up caching an HttpResponseMessage...but it has a "stream" inside it at HttpResponseMessage.Content. It's meant to be read once. Not cached. I wasn't caching a string, or some JSON, or some deserialized JSON objects, I ended up caching what's (effectively) an ephemeral one-time object and then de-serializing it every time. I mean, it's cached, but why am I paying the deserialization cost on every Page View?

The Programmer's Hindsight: This is such a classic programming/coding experience. Yesterday this was opaque and confusing. I didn't understand what was happening or why it was happening. Today - with The Programmer's Hindsight - I know exactly where I went wrong and why. Like, how did I ever think this was gonna work? ;)

As Dylan from Polly so wisely points out:

It may be more appropriate to cache at a level higher-up. For example, cache the results of stream-reading and deserializing to the local type your app uses. Which, ironically, I was already doing in my original code. It just felt heavy. Too much caching and too little business. I am trying to refactor it away and make it more generic!

This is my "ShowDatabase" (just a JSON file) that wraps my httpClient

public class ShowDatabase : IShowDatabase
{
    private readonly IMemoryCache _cache;
    private readonly ILogger _logger;
    private SimpleCastClient _client;
 
    public ShowDatabase(IMemoryCache memoryCache,
            ILogger<ShowDatabase> logger,
            SimpleCastClient client)
    {
        _client = client;
        _logger = logger;
        _cache = memoryCache;
    }
 
    static SemaphoreSlim semaphoreSlim = new SemaphoreSlim(1);
  
    public async Task<List<Show>> GetShows()
    {
        Func<Show, bool> whereClause = c => c.PublishedAt < DateTime.UtcNow;
 
        var cacheKey = "showsList";
        List<Show> shows = null;
 
        //CHECK and BAIL - optimistic
        if (_cache.TryGetValue(cacheKey, out shows))
        {
            _logger.LogDebug($"Cache HIT: Found {cacheKey}");
            return shows.Where(whereClause).ToList();
        }
 
        await semaphoreSlim.WaitAsync();
        try
        {
            //RARE BUT NEEDED DOUBLE PARANOID CHECK - pessimistic
            if (_cache.TryGetValue(cacheKey, out shows))
            {
                _logger.LogDebug($"Amazing Speed Cache HIT: Found {cacheKey}");
                return shows.Where(whereClause).ToList();
            }
 
            _logger.LogWarning($"Cache MISS: Loading new shows");
            shows = await _client.GetShows();
            _logger.LogWarning($"Cache MISS: Loaded {shows.Count} shows");
            _logger.LogWarning($"Cache MISS: Loaded {shows.Where(whereClause).ToList().Count} PUBLISHED shows");
 
            var cacheExpirationOptions = new MemoryCacheEntryOptions();
            cacheExpirationOptions.AbsoluteExpiration = DateTime.Now.AddHours(4);
            cacheExpirationOptions.Priority = CacheItemPriority.Normal;
 
            _cache.Set(cacheKey, shows, cacheExpirationOptions);
            return shows.Where(whereClause).ToList(); ;
        }
        catch (Exception e)
        {
            _logger.LogCritical("Error getting episodes!");
            _logger.LogCritical(e.ToString());
            _logger.LogCritical(e?.InnerException?.ToString());
            throw;
        }
        finally
        {
            semaphoreSlim.Release();
        }
    }
}
 
public interface IShowDatabase
{
    Task<List<Show>> GetShows();
}

I'll move a bunch of this into some generic helpers for myself, or I'll use Akavache, or I'll try another Polly Cache Policy implemented farther up the call stack! Thanks for reading my ramblings!


Sponsor: SparkPost’s cloud email APIs and C# library make it easy for you to add email messaging to your .NET applications and help ensure your messages reach your user’s inbox on time. Get a free developer account and a head start on your integration today!



© 2018 Scott Hanselman. All rights reserved.
     

Adding Resilience and Transient Fault handling to your .NET Core HttpClient with Polly

Last week while upgrading my podcast site to ASP.NET Core 2.1 and .NET. Core 2.1 I moved my Http Client instances over to be created by the new HttpClientFactory. Now I have a single central place where my HttpClient objects are created and managed, an…

b30f5128-181e-11e6-8780-bc9e5b17685eLast week while upgrading my podcast site to ASP.NET Core 2.1 and .NET. Core 2.1 I moved my Http Client instances over to be created by the new HttpClientFactory. Now I have a single central place where my HttpClient objects are created and managed, and I can set policies as I like on each named client.

It really can't be overstated how useful a resilience framework for .NET Core like Polly is.

Take some code like this that calls a backend REST API:

public class SimpleCastClient

{
private HttpClient _client;
private ILogger<SimpleCastClient> _logger;
private readonly string _apiKey;

public SimpleCastClient(HttpClient client, ILogger<SimpleCastClient> logger, IConfiguration config)
{
_client = client;
_client.BaseAddress = new Uri($"https://api.simplecast.com");
_logger = logger;
_apiKey = config["SimpleCastAPIKey"];
}

public async Task<List<Show>> GetShows()
{
var episodesUrl = new Uri($"/v1/podcasts/shownum/episodes.json?api_key={_apiKey}", UriKind.Relative);
var res = await _client.GetAsync(episodesUrl);
return await res.Content.ReadAsAsync<List<Show>>();
}
}

Now consider what it takes to add things like

  • Retry n times - maybe it's a network blip
  • Circuit-breaker - Try a few times but stop so you don't overload the system.
  • Timeout - Try, but give up after n seconds/minutes
  • Cache - You asked before!
    • I'm going to do a separate blog post on this because I wrote a WHOLE caching system and I may be able to "refactor via subtraction."

If I want features like Retry and Timeout, I could end up littering my code. OR, I could put it in a base class and build a series of HttpClient utilities. However, I don't think I should have to do those things because while they are behaviors, they are really cross-cutting policies. I'd like a central way to manage HttpClient policy!

Enter Polly. Polly is an OSS library with a lovely Microsoft.Extensions.Http.Polly package that you can use to combine the goodness of Polly with ASP.NET Core 2.1.

As Dylan from the Polly Project says:

HttpClientFactory in ASPNET Core 2.1 provides a way to pre-configure instances of HttpClient which apply Polly policies to every outgoing call.

I just went into my Startup.cs and changed this

services.AddHttpClient<SimpleCastClient>();

to this (after adding "using Polly;" as a namespace)

services.AddHttpClient<SimpleCastClient>().

AddTransientHttpErrorPolicy(policyBuilder => policyBuilder.RetryAsync(2));

and now I've got Retries. Change it to this:

services.AddHttpClient<SimpleCastClient>().

AddTransientHttpErrorPolicy(policyBuilder => policyBuilder.CircuitBreakerAsync(
handledEventsAllowedBeforeBreaking: 2,
durationOfBreak: TimeSpan.FromMinutes(1)
));

And now I've got CircuitBreaker where it backs off for a minute if it's broken (hit a handled fault) twice!

I like AddTransientHttpErrorPolicy because it automatically handles Http5xx's and Http408s as well as the occasional System.Net.Http.HttpRequestException. I can have as many named or typed HttpClients as I like and they can have all kinds of specific policies with VERY sophisticated behaviors. If those behaviors aren't actual Business Logic (tm) then why not get them out of your code?

Go read up on Polly at https://githaub.com/App-vNext/Polly and check out the extensive samples at https://github.com/App-vNext/Polly-Samples/tree/master/PollyTestClient/Samples.

Even though it works great with ASP.NET Core 2.1 (best, IMHO) you can use Polly with .NET 4, .NET 4.5, or anything that's compliant with .NET Standard 1.1.

Gotchas

A few things to remember. If you are POSTing to an endpoint and applying retries, you want that operation to be idempotent.

"From a RESTful service standpoint, for an operation (or service call) to be idempotent, clients can make that same call repeatedly while producing the same result."

But everyone's API is different. What would happen if you applied a Polly Retry Policy to an HttpClient and it POSTed twice? Is that backend behavior compatible with your policies? Know what the behavior you expect is and plan for it. You may want to have a GET policy and a post one and use different HttpClients. Just be conscious.

Next, think about Timeouts. HttpClient's have a Timeout which is "all tries overall timeout" while a TimeoutPolicy inside a Retry is "timeout per try." Again, be aware.

Thanks to Dylan Reisenberger for his help on this post, along with Joel Hulen! Also read more about HttpClientFactory on Steve Gordon's blog and learn more about HttpClientFactory and Polly on the Polly project site.


Sponsor: Check out JetBrains Rider: a cross-platform .NET IDE. Edit, refactor, test and debug ASP.NET, .NET Framework, .NET Core, Xamarin or Unity applications. Learn more and download a 30-day trial!



© 2018 Scott Hanselman. All rights reserved.