Advent of Learning 2021

I came across Advent of Code a few years ago and last year I introduced it to my team at Superdrug. For those who aren't familiar, it is a series of programming puzzles acros the 25 days of Advent; there are two puzzles each day, completing the first one unlocks the second one which is a variation or extension of the first. You can complete them in any language/environment.

This is what I've learned this year from doing AoC.

I had an aspiration this year to do at least some of the puzzles in JavaScript instead of C# so I could brush up on my JS a little bit/learn a JS unit test framework. But by the time 1st Dec rolled around I had nothing in place (or more accurately, I said I wanted to do some in JS but then did nothing else about it), so I've done everything in C# again this year, JS will have to wait another year.

I did a couple of the early days in bed first thing in the morning on my Surface Pro tablet, but I found the Type Cover keyboard a little too fiddly on an unstable surface so switched after that to doing them at my desk on a full-size keyboard.

Exponentials and Overflows

Day 6 featured exponential growth of lanternfish. If there's one thing we should have all learned in the past two years, it's how to model exponential growth... There was a trap in the puzzle in that you could complete Part 1 by modelling each individual fish, but the numbers of fish were so large in Part 2 that this approach wasn't viable (I briefly contemplated spinning up a high performance VM on Azure to run my solution...).

So I completely rewrote my solution to work on a Dictionary<int, int>, where for each day that passed the number of fish at each stage of their life dropped down a step. I ran this solution for Part 1 to confirm it gave me the same answer as my previous solution, then ran it for Part 2 and entered the result to the AoC site to get my confirmation. To my surprise the answer was still too low! Debugging into my solution I discovered some of the values in the dictionary were becoming negative numbers, and it took me a minutes thought to work out that this was probably integer overflow, the first time I can recall coming across it in 25 years of coding professionally (I very rarely deal with numbers large enough that the problem arises). I changed the dictionary to Dictionary<int, long> and the puzzle  was solved. 

Stacks

In Day 10's puzzle, you had to find a corrupt character in a string, which in another first for me marked the first time I can remember having to use a Stack. 

(I was watching my friend Dylan doing this one, and he figured out it was a Stack machine within a few seconds, but says that is the result of having had to write a LISP interpreter at college...)

A* and Djikstra

Day 15 required you to find the lowest risk path through a cave filled with chitons. Which in turn required me to learn about the Djikstra and A* algorithms, and then implement one of them. I found a C# version of A* that I could crib from, and wrote my solution. Which didn't (and at time of writing still doesn't) work.

However discussing it with my team, we were talking about practical uses for A* and think we could use it for planning routes through our stores when in-store staff are picking customer orders. 

LINQ

The puzzle for Day 7 involved crabs in tiny submarines moving from side to side and finding the most efficient point for them to align on (you get used to this sort of imagery when you do AoC...). For which based on the example I took to mean finding the point on which the most submarines were located. Which meant  I had to properly learn how to use the GroupBy method in Linq. And this was useful again on Day 14 where having constructed a long string of letters you had to find the most and least common occurring letters, I used GroupBy to project each letter and it's count, followed by OrderBy and OrderByDescending to get the two letters. 

On Day 4, we were playing bingo with a giant squid, and the task in Part 2 was to find the last winning bingo card from a set of cards. Which I was struggling with in using Enumerable.All, when I switched it to List.TrueForAll it worked first time. I've just looked up the docs for Enumerable.All and I have a feeling I may not have been calling it the correct way. (Incidentally our dev manager had a great hack for this day where after laying out the bingo cards, he transposed the columns into extra rows so that when checking for a winning card he only had to write a method for finding a winning row and not worry about columns as well.)

Also in the aforementioned C# implementation of the A* algorithm, it uses the List.FindIndex method which is not one I've come across before, and when discussing this with my colleagues they mentioned the List.IndexOf method. These two methods look pretty much identical, but it looks like their implementations are different - there's some discussion in the answers to this SO question which suggests that IndexOf is about 10 times faster than FindIndex.

Wrap Up

I've picked up quite a few things this year that I can take back to the day job, GroupBy being probably the one I'm most pleased about and potentially the most applicable. And I did set a new PB this year.

I also made some useful changes to my AoC project template. For next year I'd really like to make this a template in Visual Studio so I can have solutions named for the day they are for instead of multiple AdventTemplate solutions. And I still want to get a template together for JavaScript ahead of AoC 2022.

Regional Settings in Azure App Services

Thought I should share something we've come across this week while testing our API on Azure...

I was testing some reporting functions where we put in To and From dates, and I entered 01/01/2021 and 31/01/2021 - and I got an exception. Looking it up in AppInsights, it was an InvalidFormatException coming from DateTimeOffset.Parse. I checked the same dates on our production site (currently hosted on-prem) and it was fine. So what's the issue?

Well, it turns out that for an App Service instance, the regional settings (all those currency symbols and, most relevantly here, date formats) are set to US English, and trying to parse 31 as a month doesn't work. You can demonstrate this for yourself if you go to an App Service instance and then go to the Kudu debug console - in Powershell the command Get-WinSystemLocale will show you what your current region settings are set to. Here's a screenshot from one of mine which is located in the UK West Azure region:

So how do you fix it? There is a counterpart Set-WinSystemLocale Powershell command, which you might be tempted to try, but it won't work as you need to be an administrator on the computer, which you aren't...

If you are using the .NET framework, there is a relatively simple fix: in your web.config in the system.web element, add a new globalization element:

<system.web>

    <globalization culture="en-GB"/>

</system.web>

(Or replace the region code with the one relevant to you - there's a good list of codes here). And then your API will parse dates correctly.

In .NET Core/.NET 5, you can set the CultureInfo for the current thread in the Configure method of your Startup class as laid out in this SO answer.

You can add an application setting to your app (WEBSITE_TIME_ZONE/TZ depending on whether you are using a Windows host or a Linux host) that manages which timezone it will use, and it would be neat if there was an equivalent that allowed to specify what regional settings you want to use.

AZ-204 Study Materials

 I took - and passed :-) - the AZ-204 Developing Solutions for Azure exam yesterday. 

Study materials I used were:

• The MS learning paths listed on the exam page
• The Pluralsight exam prep path
• In particular I found the Exam Alert sessions really useful, and also even though I've been doing MS exams on and off for 20 years, I found Matt Kruzcek's session on Preparing To Take The AZ-204 exam really useful the night before as a refresher on what to expect on the day, and usefully on the question types used on the exam
• Before Xmas through work we had an invite to a Microsoft exam preparation session which walked through each entry in the list of skills measured in the exam; I picked up some really useful bits from this session so if you can get on one of them I'd recommend it

What Next?

I'm now debating what to study for next - I want to do the Azure Solutions Architect certification, but also (now that I've unlocked it) the DevOps Engineer Expert certification is likely to be both useful and relevant for work. So I'm not sure which one to work on...

10 VS Extensions You Might Have Missed

I presented a lightning talk tonight at DotNetOxford on '10 VS Extensions You Might Have Missed' covering some perhaps lesser-known items in the Visual Studio Marketplace. All of these extensions are free, and they all provide a useful addition to what you get out of the box in Visual Studio.

UPDATE: And now you can see the whole evening's set of talks at https://www.youtube.com/playlist?list=PL4qgjzgv2UYTTCbWELKjyJrcKMz02K7DH

Customize Visual Studio Window Title

https://marketplace.visualstudio.com/items?itemName=mayerwin.RenameVisualStudioWindowTitle

Azure DevOps Status Monitor

https://marketplace.visualstudio.com/items?itemName=UtkarshShigihalliandTarunArora.VSTSStatusInspector

Viasfora

https://marketplace.visualstudio.com/items?itemName=TomasRestrepo.Viasfora

Tweaks

https://marketplace.visualstudio.com/items?itemName=MadsKristensen.Tweaks

Trailing Space Visualizer

https://marketplace.visualstudio.com/items?itemName=MadsKristensen.TrailingWhitespaceVisualizer

Web Essentials

https://marketplace.visualstudio.com/items?itemName=MadsKristensen.WebEssentials2019

Const Visualiser

https://marketplace.visualstudio.com/items?itemName=MattLaceyLtd.ConstVisualizer

Warn About TODOs

https://marketplace.visualstudio.com/items?itemName=MattLaceyLtd.WarnAboutTODOs

Snippet Designer

https://marketplace.visualstudio.com/items?itemName=vs-publisher-2795.SnippetDesigner

Stack Trace Explorer

https://marketplace.visualstudio.com/items?itemName=SamirBoulema.StackTraceExplorer

I hope everyone got something out of this and found at least one extension that might be useful for them!

ASP.NET Bundling and http 403 Errors

 So I'm probably late to the game in running into this, but it bit me at work last week and I thought it was worth writing up...

I made a change to one of our internal websites that involved bringing in FontAwesome so I could use some of it's widgets. I downloaded the FontAwesome CSS and font files to serve them from our internal server, and I added them to my project under the Content folder, and I created a new StyleBundle that referenced the CSS file so I could get that sweet minification. I ran up the site on my laptop and there were my new widgets. Cool. I committed and pushed my changes, had my pull request accepted, and Azure DevOps deployed the new build to the server. I ran it up on the server - and my widgets weren't showing. But, but, but, it Works On My Machine. Time for some investigation.

My first stop was to look at the console in Chrome, where there was an important clue:

A http 403 error? But where was that coming from and how did it relate to FontAwesome? Step 2 in my investigation, look at the raw request in Fiddler and see where the 403 is coming from.

So, the first request produces a 301 result, which for those of you who haven't memorised the http status codes is 'Moved Permanently' and provides a Location in the response headers for the browser to redirect to. Notice in the Fiddler screenshot that the 301 result is generated by a request to /Content/FontAwesome, which corresponds to the name of my bundle, but the 403 result comes from /Content/FontAwesome/ - which corresponds with the path in my project that I put the FontAwesome CSS files into. 

So, there's an issue with bundling when your bundle name matches a path in the file system, and the result is this 301-403 dance where your CSS files doesn't get loaded. And the solution is obvious - don't give your bundles a name that matches a path in the file system. In my case, I renamed my bundle from ~/Content/FontAwesome to ~/bundles/FontAwesome and all was right with the world.

Why did it work on my machine? Well, in one of those 'obvious in hindsight' things, I forgot that when you have debugging enabled in your web.config, ASP.NET bundling doesn't take effect and your files that would go through the bundling are actually served straight from the file system. When I ran my code locally without debugging, I saw the same behaviour on my machine. 

Why did I pick a bundle name that matched the file system in the first place? Simple - I was following the same pattern as the boilerplate bundling code gives you for StyleBundles:

bundles.Add(new StyleBundle("~/Content/css").Include(

                      "~/Content/bootstrap.css",

                      "~/Content/site.css"));

All the above relates to .Net Framework ASP.NET sites. What happens in ASP.NET Core and can you have the same issue? Well, if you go down the bundling route in ASP.NET Core, it works differently and your bundling code produces a file for you to include in your markup rather than a virtual directory. Which means there shouldn't be an issue of a virtual directory name clashing with a physical one. See the MSDN documentation for bundling here.

Linq vs Regex

I recently had to write a password validator as part of a technical test for a prospective employer, where three of the conditions were:

• Passwords must contain an upper case character
• Passwords must contain a lower case character
• Passwords must contain a number

Having recently had occasion to learn that you can treat a string as a collection of characters and then use Linq to run set operations across the collection, in the test I used this code:

password.Any(char.IsUpper)

password.Any(char.IsLower)

password.Any(char.IsDigit)

But I worked through the same test again yesterday as an exercise with a friend, and we used Regex instead, so the code became:

new Regex(@"[A-Z]").IsMatch(password)

new Regex(@"[a-z]").IsMatch(password)

new Regex(@"[0-9]").IsMatch(password)

Plainly either version works; I don't see myself as a great Regex developer, which is partly why I reached for the Linq solution first. But seeing the solution written both ways I got to wondering if there was a performance benefit one way or the other. Clearly, there's only one way to find out...

I wrote up a console app that would use a Stopwatch to time each operation, and I ran it over 100 iterations and reported the average number of ticks taken (I started off measuring the number of milliseconds taken but this was zero...). 

And we can see that in two cases the Regex outperforms the Linq version, and in the third it's more or less even. Job done!

But... what happens if we up the number of iterations? Here's the same code run across 1 000, 10 000 and 1 000 000 iterations.

Across more iterations, the Linq version performs faster!

So which should you use? Well, as with most things programming, it depends... It's important to remember that we're talking about differences of only a tick or two, measurements so small that your user won't notice. So with that in mind, we should think about other considerations, like our relative skill levels between Linq and Regex, and which version we find more accessible and readable. So for me, if I was making the choice I'd opt for the Linq versions; I tend to find Regex somewhat impenetrable. But you might make a different choice, and that's fine too.

If you want to try my code (and the performance might be different on your machine), my code is at https://github.com/philpursglove/LinqVsRegex

(Aside: I was very pleasantly surprised to find the tooling support for writing Regexes in Visual Studio has massively improved, when you start writing your expression now you get Intellisense that shows you some of the options, and you get colour-coding inside your expression that helps you identify the parts of your expression)

Update: After I posted this, Steve got in touch to say that compiled Regexes may offer a performance benefit. You specify that a Regex is compiled by adding an option into the constructor:

new Regex("[A-Z]", RegexOptions.Compiled)

When a Regex is compiled, it is converted to MSIL and executed by the JIT compiler, and the MSIL is then cached by the regular expression engine. The price you pay for this is a longer startup time as the compilation happens, but then you get a faster execution. (Microsoft has a whole article on best practices for Regexes that covers this and some other considerations, and there's another piece about compilation and reuse of Regexes).

So let's kick the tyres on this... I added a switch to my code that allows you to switch compilation on and off. Here's some numbers over 100 iterations with compilation on.

We can see immediately that the startup impact of compiling the Regex adds an order of magnitude of overhead to our code (but remember that the practical impact of this is probably still only a matter of milliseconds), and over a relatively small number of iterations this means it will be significantly outperformed by the Linq solution. If we up the number of iterations, does paying that upfront cost return us any kind of performance dividend? Here's the same code over 1 000 000 iterations.

Over more iterations, the upfront cost of compiling the Regex gets amortised down, but (on average) it still gets beaten by Linq (and I've also tried it over 10m and 100m iterations and this pattern remains consistent). 

So Linq is consistently better, yes? Maybe, maybe not... For one thing, the Regexes I'm using are pretty simple - I have a sneaking suspicion that a compiled, more complex Regex would turn out to beat a number of chained Linq expressions. Some of the other options for Regex may also have an impact.

Steve also pointed me to Benchmarkdotnet to look at other aspects such as memory usage to get a fuller picture not only of which method looks better from the raw timings but also in terms of memory usage etc, which I may look at in a future post...

Controlling Your Costs With Azure Policy

I made a mistake. I've been trying to figure something out in Azure, and I needed a website to do it with, so I spun one up on a new App Service Plan. I missed that the default plan is a premium plan, and I only noticed this the other day. In the meantime, my site has been racking up charges, and as a result my Azure bill for July is going to be ... about £100. 

Which is (kind of) fine, experience is the best teacher, it's my mistake and I'm going to own it. It's not a problem-causing amount of money for me. And I'm not averse to spending that kind of money on Azure services on purpose, it's just not the sort of thing I want to be doing again by accident.

Fortunately, I know by using Azure Policy that you can put all kinds of controls in Azure to stop people doing this sort of thing and ending up costing your company lots of money, so I've put the same kind of control in for myself.

What I need is a policy that stops me creating any more premium App Service Plans, and this is not that difficult to achieve.

Policies work on an 'if-then' model, you give the 'if' clause a set of conditions and if the set of conditions matches, then the 'then' clause fires. So in my case when I'm creating a new Azure resource my conditions will be: the resource type is an App Service Plan, and the SKU name is not set to 'F1', and my action will be to stop the action i.e. prevent the resource from being created.

Policies are written in JSON, you can write them directly in the Azure Portal, there is also a VSCode extension to help you with creating them. Here's a skeleton policy:

"policyRule": {

    "if": {

      }

    },

    "then": {

      "effect": ""

    }

  }

Effects can be any of a number of values, including "deny" which disallows the action completely, "audit" which allows the action but logs that your policy is being violated, and a number of others including some that will change your action to make it comply with your policy.

So clearly the effect in my then clause needs to be "deny", so that it correctly prevents the action. Which leaves me to work out what the correct if clause is. As I said above, I have two conditions, so I'll be using the allof condition which translates to an AND operator (there is also anyof which is the equivalent of OR). My first condition is that I want the policy to act on App Service Plans, so I'll need to be looking at the type of thing that's being created. Policies use a system of aliases for types (with namespaces), there are two ways to find the alias you want. The hard(er) way is to use the Azure command-line tool to query a list of available aliases as suggested in the documentation e.g.

az provider show --namespace Microsoft.Web --expand "resourceTypes/aliases" --query "resourceTypes[].aliases[].name"

The easy way is to go through the resource creation process in the Azure portal, through to the 'Review and Create' step. Once there, don't create the resource, but download the ARM template, and then search through it for the 'resources' element; inside the resources element, look for a 'type' property, and the value of the property is the alias you want. Which tells me that for an App Service Plan, the alias I should be using is 'Microsoft.Web/serverfarms'. On to the SKU value! I need to look at a property of a serverfarms object and check whether or not the value is 'F1' (the name of the free tier). And if we go back to the command-line query and browse the results, we can see that under Microsoft.Web/serverfarms there is a sku.name property; we can use that with a 'notEquals' operator to check the value.

So our whole policy looks like this:

"policyRule": {

      "if": {

        "allof": [

          {

            "field": "type",

            "equals": "Microsoft.Web/serverfarms"

          },

          {

            "field": "Microsoft.Web/serverfarms/sku.name",

            "notEquals": "F1"

          }

        ]

      },

      "then": {

        "effect": "deny"

      }

    }

I applied this policy to my subscription, and here's what happens if I now try to create a premium service plan.

Note that the policy evaluation happens in the review stage, before the resource is created.

We've looked here at creating a basic (but useful!) Azure policy to restrict creation of a premium Azure resource, and you can apply that to your own subscriptions to help keep you from making the same mistake I did. And this is only a taste of what you can achieve with Azure Policy.

(And now I'm protected from accidentally creating any more premium App Service Plans - but I could create a premium database, or a storage account, or a VM... Hmm, maybe I need to go write some more policies...)