Jekyll2019-10-18T14:52:46+00:00http://blog.devbot.net/feed.xmlTommy LongMusings of an angry developer.Interpreting text messages as an Aspie2019-10-18T00:00:00+00:002019-10-18T00:00:00+00:00http://blog.devbot.net/interpret-text<p>I don’t typically or publicly talk about mental health, but for several years now my closest friends have openly and jovially attributed aspergers to me. It started with a conversation with my mum who jokingly remarked “you’re so aspie” in response to something I had said. I didn’t question the comment at the time, but later during a conversation with my closest friend she very quickly explained aspergers and seemed surprised I hadn’t <em>realised</em> my relationship to the syndrome.</p>
<p>Off the back of these conversations I spent a couple weeks researching the subject and came to realise that it was probably a fair, even if unofficial assessment. As I understand it, the preferred terminology is Autistic Spectrum Disorder (ASD) nowadays, but for the sake of brevity I’ll most refer to the term “aspie” for the rest of this article.</p>
<p>I now regularly bounce social interactions off of that same friend when I’m unsure about a given interaction (you’re the best, thank you C), and in turn I try to provide some suggestions and advice in respect to C’s eldest (I’ll refer to as B), whom we jokingly describe as “one of Tommy’s people” to describe her aspie attributes.</p>
<p>The latest of those conversations was looking at my process for interpreting text messages (be that an actual SMS, WhatsApp, e-mail, etc) to see if I could offer any advice. I’m 30-summit now so I’ve had a few years of experience to build up that process, and for the most part it serves me well, so hopefully it can be of use to others, whether you’re aspie or not.</p>
<p>I’ve included a pretty considerable intro so if you’re not here for that, feel free to <a href="#interpreting-text">skip to interpreting text</a> directly.</p>
<h2 id="difficulties">Difficulties</h2>
<p>For anyone that hasn’t come across aspies before (or haven’t realised you’ve done so), the symptoms are generally described as follows:</p>
<ul>
<li>Difficulty with social communication; more specifically, trouble understanding
<ul>
<li>facial expressions</li>
<li>tone of voice</li>
<li>jokes (and sarcasm in particular)</li>
<li>ambiguity (vague or abstract concepts)</li>
</ul>
</li>
<li>Difficulty with social interaction; more specifically we
<ul>
<li>seem insensitive (have little to no empathy)</li>
<li>prefer to be alone or in smaller groups</li>
<li>don’t find comfort in being with people</li>
<li>behave oddly or even inappropriately</li>
</ul>
</li>
<li>Restricted and repetitive behaviours and interests
<ul>
<li>rely on daily routines to reduce unpredictable or confusing situations</li>
<li>have rules for things that don’t fit into our routines, to achieve the same</li>
<li>obsess and develop highly-focused interests</li>
<li>often completely drop an interest (that we were previously obsessed with) in favour of a new interest</li>
</ul>
</li>
</ul>
<p>There are also some aspects of sensory sensitivity (being over or under sensitive), but if you want more details there’s plenty of <a href="https://www.autism.org.uk/about/what-is/asperger.aspx">resources online</a>. Being partially deaf myself, I find this aspect the least relatable, or at least can’t distinguish any sensory sensitivity differences between being aspie and that of being a bit deaf. That distinction however leads me to my next point, personality.</p>
<h2 id="personality">Personality</h2>
<p>I don’t <em>suffer</em> from Aspergers. Being aspie is <em>not</em> a bad thing. In fact, in many ways I think it’s a really great way to live.</p>
<p>For one, to many I can seem to have a prickly personality, or at least be a bit <em>weird</em>. Fortunately, I’m not <em>weird</em> alone, so much so someone decided to give it a label. My close friends know I’m aspie and if I lapse or do something odd it can simply be chalked up to that. If I don’t speak to my best friends for months it’s not because they’ve upset me or I don’t like them, I just don’t seek comfort from them. They mean a <em>LOT</em> to me, I rely on them <em>being there</em>, and I do try to be the good traditional friend when I can. But I’m not naturally inclined to stay in contact with them all the time.</p>
<p>My much better half, bless her, is also amazing in this regard. I don’t know we’ve ever talked about my being aspie specifically (though it’s pretty implicit), however she seems to have a limitless patience and is incredibly supportive (of me, not just being aspie). Being around large amounts of people I find <em>incredibly</em> stressful, but so long as my other half is with me, I feel like I can do it safe in the knowledge that I’ll be left alone, potentially for several days, to recover (thank you, beautiful).</p>
<p>Close friends aside though, I don’t go around telling people I’m aspie and expecting them to be supportive. I put <em>considerable</em> effort into fitting in; I’d like to think colleagues and such don’t think me (too) weird as a result. The point being, I don’t use being aspie as an <em>excuse</em>. I consider it part of my personality, and make adjustments accordingly.</p>
<p>To fit in aspies (or I, at least) spend an awful lot of time <em>pretending</em>. I feign interest, sympathy, emotion. I smile or “play nice” on queue, agree to attend gatherings I’m uncomfortable with, and repress to the best of my abilities any tells and symptoms. I spend considerable amounts of time analysing situations to try and determine what sympathetic or empthatic response is (or would have been) appropriate.</p>
<p>It feels like modern and emerging culture looks down on that approach, it seems to be more <em>“you be you and others will adapt”</em>. I’m a big fan of that culture in that I’ll do everything I can to be supportive and understanding of others, but (as a privileged white male) I’m against the inverse. I don’t expect or even want others to be supportive of “me unleashed”. To me, it’s not all that different to learning to have manners, just slightly more in-depth.</p>
<h2 id="research">Research</h2>
<p>With all that said, it means I want to fit in, or probably more specifically, avoid being in awkward situations more than is necessary. And that means developing methods and processes that help me understand inflections and intent that others might find obvious or are at least more naturally adept at, but I have to infer.</p>
<p>The most relatable example of which is when you interpret textual communication from others. This is a problem I suspect every one has, and probably serves as a good example of how an aspie interacts with others. Imagine a face to face chat was done in the same way as a text conversation - you can’t see their smiles, you can’t hear their tone of voice. To an aspie, we do see and hear those things, but struggle to interpet how that affects the message being conveyed, so in some respects we’re always having a text chat.</p>
<p>That isn’t always the case of course; I’ve spent long hours researching body language and facial expressions trying to train myself to be more observant; I don’t <em>think</em> I’m too bad at face to face conversations anymore. In fact I occasionally find myself providing insights to others who relied on natural inference and missed subtle signs - it’s possible that by using what is effectively an unbias observation I can spot people being fake or lying a bit more easily than those whom I’d consider to be very social.</p>
<p>I find subjects like psychology fascinating, and strive to understand not only why people are the way they are, but also where there are differences between a typical thought process and my own. I’d be lying if I said I haven’t once wondered “how do people live like that” when constrasting a <em>typical</em> process to my own, but ultimately I see the differences as just that. People are different, and being aspie is neither better nor worse.</p>
<p>It’s also important to note that I’m not an expert on this or any other subject. I’ve no formal education so everything herein is my own opinion, experience, and speculation. I’d be very keen to hear other peoples opinions or simply be told I’m wrong on a given subject (so I can study it further and correct my misunderstandings).</p>
<h2 id="interpreting-text">Interpreting Text</h2>
<p>First and foremost I want to state that there is no <em>right answer</em> to this problem as far as I’m aware. You can only understand the variables and possiblilties built largely on assumptions and then make a best effort guess. That said, you don’t have to blindly react to text-based messages.</p>
<h3 id="context">Context</h3>
<p>I’m listing this aspect first because evaluating context can change all layers thereafter. For me, context has two key aspects, the first being <em>“cause”</em>. Are you receiving the message because you did something, because you sent something, because you are a point of contact for a particular thing?</p>
<p>Cause isn’t always present or more accurately the cause can be misinformed. If someone sends me a software developer’s CV or discount codes for spectacles, it’s be<strong>cause</strong> I’ve landed on some mailing list somewhere. However, more often than not, you’re receiving a message in response to something you’ve sent to someone, in response to something you’ve published, or in response to something you have done.</p>
<p>The second crucial piece of context is what you know of the sender. If the person is a complete stranger, even the entitled denizens of the internet have a somewhat limited context. They can only be responding to something public, which can make evaluating <em>cause</em> simpler and will also limit some of the aspects we evaluate later.</p>
<p>Friends, associates, colleagues, and peers. They are the hardest to analyse in my opinion. Your knowledge of this group’s personality is limited, and they have a broader scope of things in which be responding to.</p>
<p>Finally, there’s your closest friends and/or family whom you can hopefully consider to be in a <em>safe zone</em>. If I do or say something inappropriate with this group they won’t send me some snarky hard to understand remark, they’ll explain and help me understand.</p>
<h3 id="language">Language</h3>
<p>The next step, once you’ve established a context, is to look at the language of the message itself. There is no single aspect you can look at, so you’re looking to build upon your context here but avoid making a decision.</p>
<p>If while reading through a message it starts to make you angry, sad, or any real emotion, you should <strong>stop</strong>. Easier said than done, but I highly recommend finding a mechanism that allows you to walk away, focus on something else, or do anything you can to put the message out of your mind. To effectively analyse a message you need to have a clear head. If you’ve finished the entire process and then find yourself responding emotionally then fair enough, but try to avoid it on your first pass.</p>
<p>To help clarify this, consider the following potential response to this article:</p>
<blockquote>
<p><em>Them:</em> Great article bro</p>
</blockquote>
<p>I could read that and be happy that my article has been read favourably. I might respond to that with</p>
<blockquote>
<p><em>Me:</em> That’s so kind! Thank you!!</p>
</blockquote>
<p>only to be slammed</p>
<blockquote>
<p><em>Them:</em> I was being sarcastic you turd</p>
</blockquote>
<p>The inverse could also have been true though, my intial response being</p>
<blockquote>
<p><em>Me:</em> I didn’t force you to read it, bro!</p>
</blockquote>
<p>Only to be told</p>
<blockquote>
<p><em>Them:</em> What!? I was trying to be nice!!!</p>
</blockquote>
<p>The point of following this process of waying up the variables is to avoid these misunderstandings.</p>
<h3 id="grammar">Grammar</h3>
<p>I’m not expert with grammar - I suspect there are countless mistakes in this article. However, that doesn’t mean I can’t try to understand a weigh up the grammar used by others. For example, proper use of punctuation or using an <a href="https://www.grammarly.com/blog/what-is-the-oxford-comma-and-why-do-people-care-so-much-about-it/">Oxford Comma</a> <em>might</em> be indicative of the sender being better educated or taking time to write their message. A message being heavy in emoticons <em>might</em> indicate a younger person.</p>
<p>The goal here isn’t to <em>judge</em> the sender, but instead consider the likelihood of scenarios, and where appropriate include those scenarios in your final analysis.</p>
<p>For example, I once worked with a Project Manager who seemed to find a reason to include repeated punctuation in all their e-mails, so I might receive</p>
<blockquote>
<p>Why are we doing this?????????????</p>
</blockquote>
<p>To be honest, it drove me a bit crazy getting messages like this. But when analysing this message I can neither infer that the person is angry or joking on this alone. Only that whichever of those I decide is most likely is being emphasised <del>poorly</del>. <em>/cough</em></p>
<h3 id="consequence">Consequence</h3>
<p>The final piece of this puzzle is to consider the consequences of responding to a message in a given manner. The aforementioned entitled elite of the internet (and myself included in this at times) are very poor at doing this.</p>
<p>However, when interacting with strangers in a non-professional situation, consequences are likely to be limited. The only reason to not offend a stranger, more often than not, is because you morally shouldn’t. I’m not sure whether it an aspie trait or not, but I don’t often feel guilt so if I accidentally or inadvertently offend someone, I don’t inherently <em>care</em>. I try to be a good person usually, so I may try to rectify that situation, but I’ll do so out of discipline, not because I feel bad (and yes, I appreciate that sounds worryingly sociopathic).</p>
<p>If you lack both the guilt and discipline to right the wrongs of offending someone, you should probably take a look at some of the cases of public outrage on social media. That entitlement works both ways and there are cases aplenty where individuals have been <em>set upon</em> for accidents and mis-interpretations.</p>
<p>When interacting with people you know, the consequences are usually a bit more immediate or obvious. If you call your boss an idiot, you’re probably going to end up chatting to HR. You have a much better context for those you know, so use that to determine plausible outcomes to your responses.</p>
<h3 id="analyse">Analyse</h3>
<p>If you’ve followed the above, you should have all the pieces you need, or at least, all the pieces that are likely available to you. There is no equation to sum up all the pieces and determine the best outcome, and even if there were, it would only give you a <em>likely</em> intent.</p>I don’t typically or publicly talk about mental health, but for several years now my closest friends have openly and jovially attributed aspergers to me. It started with a conversation with my mum who jokingly remarked “you’re so aspie” in response to something I had said. I didn’t question the comment at the time, but later during a conversation with my closest friend she very quickly explained aspergers and seemed surprised I hadn’t realised my relationship to the syndrome.Dependency Injection in MonoGame2019-01-06T00:00:00+00:002019-01-06T00:00:00+00:00http://blog.devbot.net/monogame-di<p>In the <a href="http://blog.devbot.net/game-loop/">previous article</a> of this <a href="http://blog.devbot.net/tag/game-loop/">mini-series</a> I talked about the <a href="http://blog.devbot.net/game-loop/#dependency-injection">problems</a> inherent in <a href="http://www.monogame.net/">MonoGame</a> (and other Game Engines) that effectively prevent Dependency Injection (DI). During the development of <a href="http://blog.devbot.net/clean-space-introduction">Clean Space</a> I made several attempts to resolve this particular issue, and in this article I want to describe the first (pretty crude) mechanism I put together.</p>
<p><img src="../images/yuck.jpg" alt="yuck" /></p>
<p>I’ll also explore (and <em>prove</em>) the performance considerations of making changes like this to MonoGame, whilst contrasting this <em>dependency injection</em> pattern to the many <a href="http://blog.ploeh.dk/2010/02/03/ServiceLocatorisanAnti-Pattern/"><em>service locator</em></a> anti-patterns I’ve seen used as workarounds.</p>
<h2 id="first-world-problems">First World Problems</h2>
<p>So the issue with MonoGame, and I’ll keep reiterating that it’s not limited to <em>just</em> MonoGame (don’t get me started on <em>Unity</em>), is that there are no abstractions, and many classes have inter-dependencies. That is to say that class <em>A</em> depends on class <em>B</em>, and class <em>B</em> depends on <em>A</em>. This problem very quickly rears it’s ugly head if you follow the application bootstrapping code, as demonstrated in the previous post.</p>
<p>Throughout this article I’ll be working exclusively with <a href="http://community.monogame.net/t/monogame-3-7-1-release/11173">v3.7.1</a> of MonoGame, but the <a href="https://docs.microsoft.com/en-us/windows/uwp/get-started/universal-application-platform-guide"><em>Universal Windows Platform</em></a> (or ‘UWP’ for short) XAML project is much the same in the latest <a href="https://github.com/MonoGame/MonoGame/tree/develop">‘dev’ branch</a> if you’re inclined to be a bit more <em>cutting edge</em>. As such, let’s recap and explore the application startup of a UWP XAML project.</p>
<blockquote>
<p><em>Note: <a href="https://msdn.microsoft.com/en-us/library/cc295302.aspx">‘XAML’</a> stands for ‘e<strong>X</strong>tensible <strong>A</strong>pplication <strong>M</strong>arkup <strong>L</strong>anguage’ and is an XML-based language designed by Microsoft to describe the visual components of your Application.</em></p>
</blockquote>
<h2 id="startup-xaml">Startup (XAML)</h2>
<p>Each platform MonoGame supports has a slightly different startup mechanism, primarily dictated by the Windows and .NET Framework calls that need to be made in order to get an application up and running. In the case of a UWP XAML application, the first bit of code to run (that we can see) is that contained in <code class="highlighter-rouge">App.xaml.cs</code>.</p>
<p>Whilst this class contains around 130 lines of code by default (from the project template) most of that isn’t too important with regards to Dependency Injection. The only line that really matters to us right now is:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="n">rootFrame</span><span class="p">.</span><span class="nf">Navigate</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">GamePage</span><span class="p">),</span> <span class="n">e</span><span class="p">.</span><span class="n">Arguments</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<blockquote>
<p><em>Note: Whilst it might seem I’m ripping into MonoGame for their lack of DI support in this article, it’s worth mentioning that Microsoft are no better - they provide basically no DI support in UWP itself.</em></p>
</blockquote>
<p>During startup, because nothing has been navigated to or displayed yet, the application will transition to <code class="highlighter-rouge">GamePage</code>. At this point, the code in <code class="highlighter-rouge">GamePage.xaml.cs</code> will be executed:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">sealed</span> <span class="k">partial</span> <span class="k">class</span> <span class="nc">GamePage</span> <span class="p">:</span> <span class="n">Page</span>
<span class="p">{</span>
<span class="k">readonly</span> <span class="n">Game1</span> <span class="n">_game</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">GamePage</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nf">InitializeComponent</span><span class="p">();</span>
<span class="c1">// Create the game.</span>
<span class="kt">var</span> <span class="n">launchArguments</span> <span class="p">=</span> <span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">;</span>
<span class="n">_game</span> <span class="p">=</span> <span class="n">MonoGame</span><span class="p">.</span><span class="n">Framework</span><span class="p">.</span><span class="n">XamlGame</span><span class="p"><</span><span class="n">Game1</span><span class="p">></span>
<span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="n">launchArguments</span><span class="p">,</span> <span class="n">Window</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">CoreWindow</span><span class="p">,</span> <span class="n">swapChainPanel</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This is the first point at which MonoGame gets involved. Here we can see that it is trying to load the <code class="highlighter-rouge">Game1</code> class into a <code class="highlighter-rouge">SwapChainPanel</code> (which the project template has declared for you in <code class="highlighter-rouge">GamePage.xaml</code>).</p>
<p>Before we take a look at the <code class="highlighter-rouge">Game1</code> class, let’s have a look at the code that executes within the <a href="https://github.com/MonoGame/MonoGame/blob/v3.7.1/MonoGame.Framework/WindowsUniversal/XamlGame.cs#L28-L31"><code class="highlighter-rouge">XamlGame<Game1>.Create</code> method</a>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="k">static</span> <span class="k">public</span> <span class="n">T</span> <span class="nf">Create</span><span class="p">(</span><span class="kt">string</span> <span class="n">launchParameters</span><span class="p">,</span> <span class="n">CoreWindow</span> <span class="n">window</span><span class="p">,</span> <span class="n">SwapChainPanel</span> <span class="n">swapChainPanel</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nf">Create</span><span class="p">(()</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">T</span><span class="p">(),</span> <span class="n">launchParameters</span><span class="p">,</span> <span class="n">window</span><span class="p">,</span> <span class="n">swapChainPanel</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This version of the <code class="highlighter-rouge">Create</code> method was actually introduced by <a href="https://www.fbonizzi.it/">Francesco Bonizzi</a> in <a href="https://github.com/MonoGame/MonoGame/commit/ebcac5888d71069b67c6005a319490ce9dba91c7#diff-75d9e8e55eb0fd7e415e540939e8087b">October 2018</a>. This greatly simplifies (though doesn’t solve) our DI considerations when compared to earlier versions. You see, whilst by default your Game class will be instantiated with <code class="highlighter-rouge">new T()</code>, we can see that there is an overloaded method of the <code class="highlighter-rouge">Create</code> method which accepts a <code class="highlighter-rouge">Func<T></code>.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
</pre></td><td class="rouge-code"><pre><span class="k">static</span> <span class="k">public</span> <span class="n">T</span> <span class="nf">Create</span><span class="p">(</span><span class="n">Func</span><span class="p"><</span><span class="n">T</span><span class="p">></span> <span class="n">gameConstructor</span><span class="p">,</span> <span class="kt">string</span> <span class="n">launchParameters</span><span class="p">,</span> <span class="n">CoreWindow</span> <span class="n">window</span><span class="p">,</span> <span class="n">SwapChainPanel</span> <span class="n">swapChainPanel</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">//argument validation...</span>
<span class="c1">// Save any launch parameters to be parsed by the platform.</span>
<span class="n">UAPGamePlatform</span><span class="p">.</span><span class="n">LaunchParameters</span> <span class="p">=</span> <span class="n">launchParameters</span><span class="p">;</span>
<span class="c1">// Setup the window class.</span>
<span class="n">UAPGameWindow</span><span class="p">.</span><span class="n">Instance</span><span class="p">.</span><span class="nf">Initialize</span><span class="p">(</span><span class="n">window</span><span class="p">,</span> <span class="n">swapChainPanel</span><span class="p">,</span> <span class="n">UAPGamePlatform</span><span class="p">.</span><span class="n">TouchQueue</span><span class="p">);</span>
<span class="c1">// Construct the game.</span>
<span class="kt">var</span> <span class="n">game</span> <span class="p">=</span> <span class="nf">gameConstructor</span><span class="p">();</span>
<span class="c1">// Set the swap chain panel on the graphics mananger.</span>
<span class="k">if</span> <span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">graphicsDeviceManager</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NullReferenceException</span><span class="p">(</span><span class="s">"You must create the GraphicsDeviceManager in the Game constructor!"</span><span class="p">);</span>
<span class="n">game</span><span class="p">.</span><span class="n">graphicsDeviceManager</span><span class="p">.</span><span class="n">SwapChainPanel</span> <span class="p">=</span> <span class="n">swapChainPanel</span><span class="p">;</span>
<span class="c1">// Start running the game.</span>
<span class="n">game</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="n">GameRunBehavior</span><span class="p">.</span><span class="n">Asynchronous</span><span class="p">);</span>
<span class="c1">// Return the created game object.</span>
<span class="k">return</span> <span class="n">game</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>There’s a lot more happening here than everything else we’ve seen so far. We can ignore the argument validation (excluded for brevity above) and jump straight to <code class="highlighter-rouge">UAPGamePlatform</code> and <code class="highlighter-rouge">UAPGamePlatform</code>. ‘UAP’ is short for <a href="https://msdn.microsoft.com/en-gb/magazine/dn973012.aspx"><em>Universal Apps</em></a> which is what we called ‘UWP’ a few years ago when the concept was first introduced. The classes in question actually belong to the <code class="highlighter-rouge">XNA</code> namespace (not MonoGame) which is a game framework Microsoft stopped developing several years ago before the MonoGame community picked up the reins.</p>
<blockquote>
<p><em>Note: ‘XNA’ is not an acronym. <a href="https://channel9.msdn.com/coding4fun/blog/ANXFramework-ANXs-not-XNA-but-kind-of">Greg Duncan explained</a> that if anything, ‘XNA’ stands for ‘<strong>X</strong>NA’s <strong>n</strong>ot <strong>a</strong>cronymed’…</em></p>
<p><img src="../images/not-funny.jpg" alt="not funny" /></p>
<p>…but at least it’s one less acronym from Microsoft to try and remember.</p>
</blockquote>
<p>The <code class="highlighter-rouge">Initialize</code> method on the <code class="highlighter-rouge">UAPGameWindow</code> class is also very involving, not to mention <code class="highlighter-rouge">internal</code> so is otherwise not visible to us.</p>
<p>Once the XNA components are configured, MonoGame goes on to instantiate our <code class="highlighter-rouge">Game1</code> class (passed as a generic <code class="highlighter-rouge">T</code>), then complain (by throwing an exception) if you don’t set the <code class="highlighter-rouge">graphicsDeviceManager</code> property (which is also <code class="highlighter-rouge">internal</code>…) before calling the public <code class="highlighter-rouge">Run</code> method on the inherited <code class="highlighter-rouge">Game</code> class. Remember this bit because it’ll become important later.</p>
<p>The reason I keep highlighting these <code class="highlighter-rouge">internal</code> members is it means, unless we’re writing code in the same assembly as MonoGame, or MonoGame for some reason exposes it’s internal members to our code using an <code class="highlighter-rouge">InternalsVisibleTo</code> attribute, we can only <em>access</em> those classes via reflection, and doing so would not be type safe. This is a <em>huge</em> problem with regards to extensibility because it makes it either impossible or very <em>unsafe</em> to extend and change behaviours.</p>
<p><img src="../images/happiness-challenged.jpg" alt="not happy" /></p>
<h2 id="interdependencies">Interdependencies</h2>
<p>Of the last block of code we looked at, the biggest problem by far is interactions with the <code class="highlighter-rouge">graphicsDeviceManager</code> property on the <code class="highlighter-rouge">Game</code> class. The property looks as follows:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</pre></td><td class="rouge-code"><pre><span class="k">internal</span> <span class="n">GraphicsDeviceManager</span> <span class="n">graphicsDeviceManager</span>
<span class="p">{</span>
<span class="k">get</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_graphicsDeviceManager</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_graphicsDeviceManager</span> <span class="p">=</span> <span class="p">(</span><span class="n">IGraphicsDeviceManager</span><span class="p">)</span>
<span class="n">Services</span><span class="p">.</span><span class="nf">GetService</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">IGraphicsDeviceManager</span><span class="p">));</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">(</span><span class="n">GraphicsDeviceManager</span><span class="p">)</span><span class="n">_graphicsDeviceManager</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">set</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_graphicsDeviceManager</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">"GraphicsDeviceManager already registered for this Game object"</span><span class="p">);</span>
<span class="n">_graphicsDeviceManager</span> <span class="p">=</span> <span class="k">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>There seems to be some sort of <em>in-joke</em> about setting important fields as <code class="highlighter-rouge">internal</code> and not making them <code class="highlighter-rouge">virtual</code>, but I understand that this property is so intrinsic to other code in MonoGame that they wanted to protect it. We can see from the <em>getter</em> that if the private field <code class="highlighter-rouge">_graphicsDeviceManager</code> is <code class="highlighter-rouge">null</code> then it will populate the field using <code class="highlighter-rouge">Services</code>.</p>
<p>We haven’t talked about this <code class="highlighter-rouge">Services</code> property yet but we’ll circle back to it <a href="#services">later</a>. We can also see from the <em>setter</em> that if we make the mistake of attempting to set the <code class="highlighter-rouge">graphicsDeviceManager</code> property twice, we’ll get an <code class="highlighter-rouge">InvalidOperationException</code>.</p>
<p>With that all in mind, let’s take a look at this <code class="highlighter-rouge">GraphicDeviceManager</code> class, and in particular, it’s constructor:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="nf">GraphicsDeviceManager</span><span class="p">(</span><span class="n">Game</span> <span class="n">game</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">game</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">ArgumentNullException</span><span class="p">(</span><span class="s">"game"</span><span class="p">,</span> <span class="s">"Game cannot be null."</span><span class="p">);</span>
<span class="n">_game</span> <span class="p">=</span> <span class="n">game</span><span class="p">;</span>
<span class="c1">// sets a load of graphics properties which I'll omit</span>
<span class="c1">// Let the plaform optionally overload construction defaults.</span>
<span class="nf">PlatformConstruct</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="n">_game</span><span class="p">.</span><span class="n">Services</span><span class="p">.</span><span class="nf">GetService</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">IGraphicsDeviceManager</span><span class="p">))</span> <span class="p">!=</span> <span class="k">null</span><span class="p">)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">ArgumentException</span><span class="p">(</span><span class="s">"A graphics device manager is already registered. The graphics device manager cannot be changed once it is set."</span><span class="p">);</span>
<span class="n">_game</span><span class="p">.</span><span class="n">graphicsDeviceManager</span> <span class="p">=</span> <span class="k">this</span><span class="p">;</span>
<span class="n">_game</span><span class="p">.</span><span class="n">Services</span><span class="p">.</span><span class="nf">AddService</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">IGraphicsDeviceManager</span><span class="p">),</span> <span class="k">this</span><span class="p">);</span>
<span class="n">_game</span><span class="p">.</span><span class="n">Services</span><span class="p">.</span><span class="nf">AddService</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">IGraphicsDeviceService</span><span class="p">),</span> <span class="k">this</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>If you scroll up to the previous chapter, you may recall that <code class="highlighter-rouge">graphicsDeviceManager</code> was a property on the <code class="highlighter-rouge">Game</code> class (I told you that bit was important). During our exploration of the <code class="highlighter-rouge">Create</code> method we saw that if you don’t set that property when constructing your <code class="highlighter-rouge">Game</code> then it will throw an exception. Here’s a mini-recap:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre><span class="c1">// Construct the game.</span>
<span class="kt">var</span> <span class="n">game</span> <span class="p">=</span> <span class="nf">gameConstructor</span><span class="p">();</span>
<span class="c1">// Set the swap chain panel on the graphics mananger.</span>
<span class="k">if</span> <span class="p">(</span><span class="n">game</span><span class="p">.</span><span class="n">graphicsDeviceManager</span> <span class="p">==</span> <span class="k">null</span><span class="p">)</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">NullReferenceException</span><span class="p">(</span><span class="s">"You must create the GraphicsDeviceManager in the Game constructor!"</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>For those with experience with DI containers, you’ll immediately see the issue here. In order to construct a <code class="highlighter-rouge">Game</code>, we must create <code class="highlighter-rouge">GraphicsDeviceManager</code>, but in order to construct a <code class="highlighter-rouge">GraphicsDeviceManager</code>, we must create a <code class="highlighter-rouge">Game</code>. Depending on the DI container you use, you’ll either end up with a <code class="highlighter-rouge">StackOverflowException</code> as the container loops around playing <em>hot-potato</em> with these constructors, or the container will realise that neither class can be instantiated and throw an exception accordingly.</p>
<h2 id="submodules">Submodules</h2>
<p>There are a number of approaches to tackling the above problems (and other problems we haven’t discovered yet), but the first one I’ve opted for here is to use a <a href="https://git-scm.com/docs/git-submodule">git submodule</a> to pull in the MonoGame source code, have my project reference the submodule, and then hack away at it until things are working.</p>
<blockquote>
<p><em>Note: I don’t want to spend time explaining submodules and how to work with them, so if it’s a concept you’re not familiar or comfortable with, I recommend checking out a <a href="https://git-scm.com/book/en/v2/Git-Tools-Submodules">tutorial</a> and then heading back here afterwards.</em></p>
</blockquote>
<p>Before I get started on this approach though, I want to say that this is a <em>bad</em> idea. By submoduling the MonoGame repository I’m introducing a fork that I then become responsible for maintaining (like applying the latest commits from the parent). On a large active repository like MonoGame, it really isn’t very workable. However, it is a useful mechanism for <em>exploring</em> problems with frameworks you depend on.</p>
<p>Per the <a href="https://github.com/MonoGame/MonoGame#source-code">MonoGame README</a>, once you’ve initialised the MonoGame submodule and in turn, MonoGame’s submodules, you can run <code class="highlighter-rouge">Protobuild.exe</code> to generate the necessary <code class="highlighter-rouge">csproj</code>’s. Protobuild will generate a <code class="highlighter-rouge">MonoGame.Framework.WindowsUniversal.csproj</code> in the <code class="highlighter-rouge">MonoGame.Framework</code> directory, which we can add to our solution as an existing project:</p>
<p><img src="../images/monogame-submodule.png" alt="monogame submodule" /></p>
<blockquote>
<p><em>Note: Once you’ve submoduled MonoGame, create yourself a branch at the v3.7.1 tag and checkout there, otherwise you’ll default to the <code class="highlighter-rouge">develop</code> branch which has been set as the default.</em></p>
</blockquote>
<p>With the MonoGame source code now available, drop the <em>assembly</em> reference to <code class="highlighter-rouge">MonoGame.Framework</code> in your own application, and add a <em>project</em> reference to <code class="highlighter-rouge">MonoGame.Framework.WindowsUniversal</code> in it’s place. Your application should still build and be able to execute (though in my case I do get a bunch of build warnings from the MonoGame project which I’ll ignore).</p>
<p>Whilst that seems like a lot of work to have achieved nothing, what it actually means is that we can now edit the MonoGame code and see that reflected immediately in our own application. Given the power and authority to change MonoGame code, the next sections outline the changes I would make in order to better support dependency injection.</p>
<h2 id="initialisation">Initialisation</h2>
<p>You may remember from my <a href="http://blog.devbot.net/game-loop/#dependency-injection">previous post</a> there’s actually two aspects of dependency injection I would like to improve. The first is constructor injection for my <code class="highlighter-rouge">Game</code> class during initialisation (when the game first loads, or perhaps resumes from sleeping in some cases). The second case was <em>scoped convention injection</em> during each iteration of the loop (each time the <code class="highlighter-rouge">Update</code> method is called).</p>
<p>The two mechanisms are very different so we’ll tackle each in turn. Whenever you make a change to an application, there are several things you need to take into account, one of which is <em>performance</em>. I could spend time benchmarking application initialisation times before and after my changes, but the fact is, if your game reads any file (textures, sprites, audio clips, etc) or pops out to the internet, which is very likely when you’re loading a game, the minuscule amount of time it takes to run dependency injection by comparison makes it a complete non-entity. As such, I’m not even going to consider performance for game initialisation. However, when we look at interfering with the update loop, which runs very often and needs to execute very quickly, we’ll definitely spend some time benchmarking.</p>
<p>When following the startup code, we saw that the execution flow is roughly:</p>
<p><code class="highlighter-rouge">App.xaml.cs</code> → <code class="highlighter-rouge">GamePage.xaml.cs</code> → <code class="highlighter-rouge">Game1.cs</code> → <code class="highlighter-rouge">GraphicsDeviceManager.cs</code></p>
<p>As we discovered above, the <code class="highlighter-rouge">GraphicsDeviceManager</code> requires a reference to <code class="highlighter-rouge">Game1</code>. Normally, this would be a good thing. We call it <em><a href="http://blog.devbot.net/composition/#dependency-inversion-principle-dip">dependency inversion</a></em>, however the two classes are very <em><a href="https://stackoverflow.com/a/3085419/707618">strongly-coupled</a></em>.</p>
<p>For Application Initialisation it’s actually pretty simple to hook in a DI container if we accept the caveat that the constructor for <code class="highlighter-rouge">Game1</code> must always instantiate the <code class="highlighter-rouge">GraphicsDeviceManager</code>. To prove this, let’s install the <a href="https://www.nuget.org/packages/microsoft.extensions.dependencyinjection"><code class="highlighter-rouge">Microsoft.Extensions.DependencyInjection</code> NuGet package</a>.</p>
<blockquote>
<p><em>Note: In order to install this package, per the <a href="https://docs.microsoft.com/en-us/dotnet/standard/net-standard#net-implementation-support">NetStandard Compatibility Documentation</a>, you need to ensure your minimum UWP target is the Falls Creators Update (Windows 10.0 Build 16299).</em></p>
</blockquote>
<p>Once installed, modify <code class="highlighter-rouge">GamePage.xaml.cs</code> to read as follows:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="nf">GamePage</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nf">InitializeComponent</span><span class="p">();</span>
<span class="c1">// Create the game.</span>
<span class="kt">var</span> <span class="n">launchArguments</span> <span class="p">=</span> <span class="kt">string</span><span class="p">.</span><span class="n">Empty</span><span class="p">;</span>
<span class="c1">// create an MS service collection</span>
<span class="kt">var</span> <span class="n">services</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServiceCollection</span><span class="p">();</span>
<span class="c1">// add our game class</span>
<span class="n">services</span><span class="p">.</span><span class="n">AddSingleton</span><span class="p"><</span><span class="n">Game1</span><span class="p">>();</span>
<span class="c1">// Our game class creates the GraphicsDeviceManager, which in turn adds itself to the Game.Services property</span>
<span class="c1">// we can peal those services back out by fetching our singleton Game1 instance:</span>
<span class="n">services</span><span class="p">.</span><span class="nf">AddSingleton</span><span class="p">(</span><span class="n">provider</span> <span class="p">=></span> <span class="n">provider</span>
<span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">Game1</span><span class="p">>()</span>
<span class="p">.</span><span class="n">Services</span>
<span class="p">.</span><span class="n">GetService</span><span class="p"><</span><span class="n">IGraphicsDeviceManager</span><span class="p">>());</span>
<span class="n">services</span><span class="p">.</span><span class="nf">AddSingleton</span><span class="p">(</span><span class="n">provider</span> <span class="p">=></span> <span class="n">provider</span>
<span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">Game1</span><span class="p">>()</span>
<span class="p">.</span><span class="n">Services</span>
<span class="p">.</span><span class="n">GetService</span><span class="p"><</span><span class="n">IGraphicsDeviceService</span><span class="p">>());</span>
<span class="c1">// add your custom services here...</span>
<span class="c1">// build a microsoft di container</span>
<span class="kt">var</span> <span class="n">serviceProvider</span> <span class="p">=</span> <span class="n">services</span><span class="p">.</span><span class="nf">BuildServiceProvider</span><span class="p">();</span>
<span class="c1">// pass a delegate of our di container to the Create method overload</span>
<span class="n">_game</span> <span class="p">=</span> <span class="n">MonoGame</span><span class="p">.</span><span class="n">Framework</span><span class="p">.</span><span class="n">XamlGame</span><span class="p"><</span><span class="n">Game1</span><span class="p">>.</span><span class="nf">Create</span><span class="p">(</span><span class="n">serviceProvider</span><span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">Game1</span><span class="p">>,</span> <span class="n">launchArguments</span><span class="p">,</span> <span class="n">Window</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">CoreWindow</span><span class="p">,</span> <span class="n">swapChainPanel</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Hopefully the code comments cover the basics in that snippet. We can confirm this works by adding a dependency to our <code class="highlighter-rouge">Game1</code> constructor:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="nf">Game1</span><span class="p">(</span><span class="n">Foo</span> <span class="n">foo</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_foo</span> <span class="p">=</span> <span class="n">foo</span><span class="p">;</span>
<span class="c1">// this line must be kept despite the DI, sorry!</span>
<span class="n">graphics</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">GraphicsDeviceManager</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="n">Content</span><span class="p">.</span><span class="n">RootDirectory</span> <span class="p">=</span> <span class="s">"Content"</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Then add a <code class="highlighter-rouge">Foo</code> implementation to our DI container in the <code class="highlighter-rouge">GamePage.xaml.cs</code> class prior to the DI container being created:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre><span class="c1">// add your custom services here...</span>
<span class="n">services</span><span class="p">.</span><span class="n">AddSingleton</span><span class="p"><</span><span class="n">Foo</span><span class="p">>();</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>But wait, we now have an error in this class. The generic on the <code class="highlighter-rouge">XamlGame</code> class is declared as <code class="highlighter-rouge">where T : Game, new()</code>.</p>
<p><img src="../images/ffs.jpg" alt="ffs" /></p>
<p>Well, if you’re only after a hacky application initialisation, you can update the constructors on <code class="highlighter-rouge">Game1</code> to be as follows:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
</pre></td><td class="rouge-code"><pre><span class="c1">// this entire ctor must be kept despite the DI, sorry!</span>
<span class="k">public</span> <span class="nf">Game1</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">graphics</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">GraphicsDeviceManager</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="n">Content</span><span class="p">.</span><span class="n">RootDirectory</span> <span class="p">=</span> <span class="s">"Content"</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="nf">Game1</span><span class="p">(</span><span class="n">Foo</span> <span class="n">foo</span><span class="p">)</span> <span class="p">:</span> <span class="k">this</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">_foo</span> <span class="p">=</span> <span class="n">foo</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Because we <em>must</em> have a public parameterless constructor to keep <code class="highlighter-rouge">XamlGame</code> happy, I’ve moved the default instantiation code into it. We can then have a constructor of our choosing (such as the one taking a <code class="highlighter-rouge">Foo</code> in the example above) so long as it calls the default constructor (which I’m doing above with the <code class="highlighter-rouge">this()</code> instruction).</p>
<p>If you run the above code you’ll see that the constructor that takes a <code class="highlighter-rouge">Foo</code> is called first, which in turn immediately calls the default constructor. It’s not pretty, but unless we’re prepared to get our hands real dirty (per the next blog post in this series!!) then that’s the best you can manage.</p>
<h2 id="summary">Summary</h2>
<p>This has been a really long post for what amounts to a very sub-par solution, and that’s ignoring the fact that we’ve only tackled the much easier <em>application initialisation</em> aspect. However, it’s all a learning process and we now have all the pieces in place to enact some real change and improve this solution.</p>
<p>There are a minimum of 3 more articles to this series (one of which I’ve already started writing up). I’ve come up with a fairly <em>dirty</em> way of improving on this DI mechanism that hopefully doesn’t involve needing to make changes to MonoGame itself. Then I’ll explore modifying MonoGame (and hopefully look into pushing some changes back upstream) for Application Initialisation, and a final article on heavily modifying the engine in order to support the scoped update loop injection that I mentioned previously. As you can see, despite the length of this blog post, the journey has but begun…</p>
<p>As an aside, it took me almost a whole year to write the second article in this mini-series, and even then I only picked it up because someone showed interest on Twitter (thanks <a href="https://twitter.com/Osoy13">@Osoy13</a> for your <a href="https://twitter.com/Osoy13/status/1080971742569512960">tweet</a>). If this series interests you please drop me a note below or on twitter - it makes a big difference to my motivation to blog more!</p>In the previous article of this mini-series I talked about the problems inherent in MonoGame (and other Game Engines) that effectively prevent Dependency Injection (DI). During the development of Clean Space I made several attempts to resolve this particular issue, and in this article I want to describe the first (pretty crude) mechanism I put together.Game Loop2018-01-17T00:00:00+00:002018-01-17T00:00:00+00:00http://blog.devbot.net/game-loop<p>Quite some time ago, following a short and shallow <a href="http://blog.devbot.net/industry/">review</a> of the game development industry, I started writing a game; <a href="http://blog.devbot.net/clean-space-introduction/">Clean Space</a>. Work on the game is ongoing and I have learned an incredible amount so far. Whilst I’ll happily talk the ears off of anyone in proximity that shows but a hint of interest, I’ve done very poorly at writing down some of the lessons I’ve learned throughout the course of Clean Space’s development. So, in this article, I want to talk about <em>“The Game Loop”</em>, how it fits into major Game Engines, and why I very quickly came to the conclusion that it is <strong>wrong</strong>.</p>
<p><img src="../images/flat-earth.jpg" alt="flat earth" /></p>
<h2 id="concept">Concept</h2>
<p>Let’s start with describing what exactly the Game Loop is. Anyone that has written a game will have come across this very well known and widely adopted pattern of game development. I don’t want to spend particularly long on this because authors like <a href="https://twitter.com/munificentbob">Bob Nystrom</a> have some really <a href="http://gameprogrammingpatterns.com/game-loop.html">great content</a> on game loops, why they exist, and how they solve certain problems.</p>
<p>The premise however is something along the lines of the following:</p>
<p><img src="../images/game-loop.png" alt="game loop" /></p>
<p>Unlike event driven frameworks where code executes in response to an event being triggered, games cycle through frame by frame, each time looking for what user inputs have been registered, then processing the game state, before rendering (drawing onto the screen) said state.</p>
<p>There are of course variations to this which accommodate different usages and alleviate certain problems, but that’s the crux of it.</p>
<p>Within the game loop, access is provided in one way or another to a <em>delta time</em> which indicates for how long the game has been running. With this information, you should be able to determine the state of each entity. For example, knowing an in-game car is travelling at a given speed in a given direction, and having been provided how long the game has been running, you can calculate where exactly the car should be and draw/render it at that location.</p>
<h2 id="game-engines">Game Engines</h2>
<p>Game Engines, paraphrased from <a href="https://en.wikipedia.org/wiki/Game_engine">wiki</a> are:</p>
<blockquote>
<p>Software frameworks designed for the creation of games. The core functionality provided by game engines include rendering engines, physics engines, collision detection, sound… <em>and a whole lot more</em>.</p>
</blockquote>
<p>Long story short, they’re complicated beasts working hard to make game development as <em>accessible</em> as possible. The maths behind a lot of game mechanics, especially rendering, is mind boggling. A game engine can abstract that away and provide far simpler interfaces.</p>
<p>I work primarily in C#, so the biggest engines of relevance (and therefore the only 2 I’ll talk about throughout the rest of this article), are <a href="https://unity3d.com/">Unity</a>, and <a href="http://www.monogame.net/">MonoGame</a>.</p>
<p>Whilst the API surfaces differ wildly between these two frameworks, they both operate Game Loops. Unity hides the loop almost completely, but evidence of it is still apparent in the <a href="https://docs.unity3d.com/ScriptReference/MonoBehaviour.html">MonoBehaviour</a> class <em>“from which every Unity script derives”</em>. This class has a broad range of methods and event mechanisms you can latch onto within the execution of the loop, but the most obvious (and probably most common) of these is the <a href="https://docs.unity3d.com/ScriptReference/MonoBehaviour.Update.html"><code class="highlighter-rouge">Update</code> Method</a>.</p>
<p>The signature for the method is completely vanilla:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="k">void</span> <span class="nf">Update</span><span class="p">();</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>And access to the aforementioned <em>delta time</em> is achieved through the static <code class="highlighter-rouge">Time.deltaTime</code> (don’t get me started on the casing…).</p>
<p>MonoGame by contrast has a much more obvious loop sat in the class that derives from <a href="http://www.monogame.net/documentation/?page=T_Microsoft_Xna_Framework_Game">Game</a>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="k">void</span> <span class="nf">Update</span><span class="p">(</span><span class="n">GameTime</span> <span class="n">gameTime</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Instead of accessing a <em>delta</em> from some <code class="highlighter-rouge">static</code> member, the <a href="http://www.monogame.net/documentation/?page=T_Microsoft_Xna_Framework_GameTime"><code class="highlighter-rouge">GameTime</code></a> (correctly cased…) is passed to the update method.</p>
<p>I’ll state now that I vastly prefer the MonoGame API over Unity, not just because Unity refers to <strong>code</strong> as <em>scripts</em>, or because Unity hasn’t come across a C# convention for properly naming and casing members, nor even because most of Unity’s examples and documentation uses Java formatting conventions. It’s because I’ve found MonoGame a lot more extensible, and given the problems I highlight below, the ability to extend (and <em>correct</em>) certain aspects is very important to me.</p>
<h2 id="dependency-injection">Dependency Injection</h2>
<p>I’m not going to waste time explaining the benefits of dependency injection (DI) here. I consider it a must have, and I consider <a href="http://blog.ploeh.dk/2010/02/03/ServiceLocatorisanAnti-Pattern/">service location</a> completely unacceptable. However, neither of these very mature game engines are at all supportive of DI.</p>
<p>Unity in particular is such a black box that I didn’t really get anywhere with it. Instead, if you want DI in Unity, I suggest you check out <a href="https://github.com/modesttree/Zenject">Zenject</a>. Given my lack of affection for Unity, I’m not really interested in spending more time looking at the internal structure in order to achieve DI for myself.</p>
<p>MonoGame is a different story. The engine instantiates (or <em>bootstraps</em>) itself slightly differently depending on the platform you’re developing against. If we use <em>Universal Windows Platform</em> (UWP) as a common denominator, available as a platform in both engines, I’ll explain how MonoGame’s dependency structure, whilst at least accessible, by default would force an awkwardly positioned <a href="http://blog.ploeh.dk/2011/07/28/CompositionRoot/">composition root</a>.</p>
<p>MonoGame supports two versions of UWP. In the case of ‘UWP (Xaml)’, application initialisation is as follows <em>(note, many lines removed for sake of brevity)</em>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">class</span> <span class="nc">App</span> <span class="p">:</span> <span class="n">Application</span> <span class="c1">// App.xaml.cs</span>
<span class="p">{</span>
<span class="k">protected</span> <span class="k">override</span> <span class="k">void</span> <span class="nf">OnLaunched</span><span class="p">(</span><span class="n">LaunchActivatedEventArgs</span> <span class="n">e</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">rootFrame</span><span class="p">.</span><span class="nf">Navigate</span><span class="p">(</span><span class="k">typeof</span><span class="p">(</span><span class="n">GamePage</span><span class="p">),</span> <span class="n">e</span><span class="p">.</span><span class="n">Arguments</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">GamePage</span> <span class="p">:</span> <span class="n">Page</span> <span class="c1">// GamePage.xaml.cs</span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">GamePage</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">_game</span> <span class="p">=</span> <span class="n">MonoGame</span><span class="p">.</span><span class="n">Framework</span><span class="p">.</span><span class="n">XamlGame</span><span class="p"><</span><span class="n">Game1</span><span class="p">></span>
<span class="p">.</span><span class="nf">Create</span><span class="p">(</span><span class="n">launchArguments</span><span class="p">,</span> <span class="n">Window</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">CoreWindow</span><span class="p">,</span> <span class="n">swapChainPanel</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p><code class="highlighter-rouge">Game1</code> on line 13 above is the start of your game code. The ‘UWP (Core)’ variant is similar (though simpler):</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">static</span> <span class="k">class</span> <span class="nc">Program</span>
<span class="p">{</span>
<span class="k">static</span> <span class="k">void</span> <span class="nf">Main</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">factory</span> <span class="p">=</span> <span class="k">new</span> <span class="n">MonoGame</span><span class="p">.</span><span class="n">Framework</span><span class="p">.</span><span class="n">GameFrameworkViewSource</span><span class="p"><</span><span class="n">Game1</span><span class="p">>();</span>
<span class="n">Windows</span><span class="p">.</span><span class="n">ApplicationModel</span><span class="p">.</span><span class="n">Core</span><span class="p">.</span><span class="n">CoreApplication</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="n">factory</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>In both of these cases, I’d at the very least like to DI my <code class="highlighter-rouge">Game1</code> class. In fact, I’d even go further than that and in addition to having the Game itself constructor injected to help me with game intialisation, I’d also want the means to convention inject the <code class="highlighter-rouge">Update</code> method with a <em>frame-scoped</em> container. That is to say, I’d like to be able to do the following:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">class</span> <span class="nc">Game1</span>
<span class="p">{</span>
<span class="k">public</span> <span class="n">Task</span> <span class="nf">Update</span><span class="p">(</span><span class="n">Logger</span> <span class="n">logger</span><span class="p">,</span> <span class="n">MyService</span> <span class="n">foo</span><span class="p">,</span> <span class="n">MyOtherThing</span> <span class="n">bar</span><span class="p">)</span>
<span class="p">{</span>
<span class="c1">// have the services be scoped to the frame</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>So what’s preventing this? Well if we take a look at the constructor of <code class="highlighter-rouge">Game1</code> (auto-generated from the MonoGame templates) we’ll see the following:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="nf">Game1</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">graphics</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">GraphicsDeviceManager</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The <code class="highlighter-rouge">graphics</code> field we assign the new <a href="http://www.monogame.net/documentation/?page=T_Microsoft_Xna_Framework_GraphicsDeviceManager"><code class="highlighter-rouge">GraphicsDeviceManager</code></a> to isn’t actually used in the rest of the template, so you might think you can just throw that line of code away. You’d be wrong:</p>
<blockquote>
<p>System.NullReferenceException: ‘You must create the GraphicsDeviceManager in the Game constructor!’</p>
</blockquote>
<p><em>(Actually, in UWP Core you won’t get any exception at time of writing, the program will just terminate)</em>.</p>
<p>To further hamper matters, the <code class="highlighter-rouge">GraphicsDeviceManager</code> constructor contains the following lines:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="nf">GraphicsDeviceManager</span><span class="p">(</span><span class="n">Game</span> <span class="n">game</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_game</span> <span class="p">=</span> <span class="n">game</span><span class="p">;</span>
<span class="n">_game</span><span class="p">.</span><span class="n">graphicsDeviceManager</span> <span class="p">=</span> <span class="k">this</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>…and that <code class="highlighter-rouge">Game.graphicsDeviceManager</code> property is <code class="highlighter-rouge">internal</code>. Anyone that has done a good amount of dependency injection will tell you that two classes depending upon one another is unattainable, akin the age old adage, <em>‘chicken vs. egg’</em>.</p>
<p>We’ll consider this problem number one.</p>
<h2 id="io">I/O</h2>
<p>It basically isn’t possible to write a game that completely avoids I/O. At the very least, the game will need to be loaded from disk at some stage. However, more importantly, whilst not unavoidable, it would be extremely problematic to avoid I/O within the game loop (if you were to follow the conventions prescribed by most game engines). Any time you read or write to disk or network, you involve I/O.</p>
<p>Traditionally, I/O or read/write operations were what we called <em>blocking</em> operations. That is, a computational <em>thread</em> would have to sit and wait for the operation to complete (be blocked) until the operation completed and execution could continue. Fortunately, this problem was solved in Windows at least many years ago with the introduction of <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa365198(v=vs.85).aspx"><em>I/O Completion Ports</em></a> (or <em>IOCP</em> for short).</p>
<p>For more than 5 years now, .Net has had <a href="https://docs.microsoft.com/en-us/dotnet/csharp/async"><code class="highlighter-rouge">await/async</code> functionality</a> that makes utilising IOCP all but child’s play. And yet, looking back to the game loop signatures I described above, IOCP is at best a second class citizen of these game engines; I am yet to come across an API in either of the game frameworks that utilises <code class="highlighter-rouge">async/await</code>.</p>
<p>The reason for this, I would speculate, is the complexity of <a href="https://msdn.microsoft.com/en-us/library/system.threading.synchronizationcontext(v=vs.110).aspx"><code class="highlighter-rouge">SynchronizationContexts</code></a>. UI’s are very particular, in an attempt to protect themselves, about which thread can access and manipulate them. If a game engine were to introduce such mechanisms it would be easier for the UI thread to get lost during execution of a loop iteration, and I expect the respective forums and issue trackers would be inundated with users having sync context issues.</p>
<p>However, if you’re comfortable with these threading mechanics, it becomes a nuisance that the APIs are all but prohibitive of async usage. We’ll call this problem number two.</p>
<h2 id="physics">Physics</h2>
<p>There’s many aspects to physics, so I’ll just pick on a couple of obvious collision based examples that you may well have come across whilst playing games. Imagine that all your game logic is tied into your game loop which iterates <em>per frame</em>. We’ll start with the example of a ball bouncing off of a wall:</p>
<p><img src="../images/collision-01.png" alt="clean collision" /></p>
<p>In this example we see in the first frame that the blue ball is approaching a red wall. In frame two, the ball hits the wall, and in frame three the ball has bounced off of the wall. Easy.</p>
<p>But what if the ball is moving too fast for your frame rate and the exact point at which the ball hits the wall is missed?</p>
<p><img src="../images/collision-02.png" alt="intersecting collision" /></p>
<p>As before, we start with a ball approaching the wall. But, in the second frame the ball is calculated to be intersecting the wall! So, what should the physics engine do about this? It knows that two solid objects can’t intersect, so in frame 3, this arbitrary examples applies sufficient force to the ball that it flies off into the sunset. You see examples of this in modern games very often, physics engines over-compensating for overlapped bounding boxes.</p>
<p>There is also a third simple example to consider:</p>
<p><img src="../images/collision-03.png" alt="bypassed collision" /></p>
<p>In this example, the ball was moving so fast (or the frame rate so low) that in frame 1 it appears to the right of the wall, and in frame 2 it’s on the left. There was no frame where the ball and wall were in contact, so no collision was acted upon. The ball simply passes through. This example is less common, though probably more as a result of superior hardware allowing high frame rates as opposed to polished physics engines!</p>
<p>We’ll call this problem three.</p>
<h2 id="performance">Performance</h2>
<p>Have you ever played a game that completely freezes up when you click <em>“Load Game”</em>, or more obnoxiously as the message <em>“Autosaving…”</em> appears? Have you ever clicked a button or passed a way point, and the world just stops?</p>
<p>This is a symptom of performing operations on the UI thread that do not belong there. Actions like those mentioned earlier in the <a href="#io">I/O section</a>.</p>
<p>However, if you attempt to alleviate this by performing all your game logic processing on one thread and leaving the UI thread to do nothing more than render that state, you have to <em>synchronise</em> those threads. That’s a non-trivial task; you never want more than one thread accessing the same thing, at the same time.</p>
<p>For example, imagine you have a ball that you want to move diagonally. The ball starts in <code class="highlighter-rouge">X = 0; Y = 0</code> and you want to move it to <code class="highlighter-rouge">X = 1; Y = 1</code>.</p>
<p>To start with, the ball is at zero and the UI thread renders it so. Your game state thread then begins moving your ball, but only gets as far as setting <code class="highlighter-rouge">X = 1</code> before the UI thread jumps in and reads the balls state, which at this point in time is <code class="highlighter-rouge">X = 1, Y = 0</code>. Your ball suddenly hops up instead of diagonally on the screen…</p>
<p>Thread synchronisation has actually been around for longer than there has been multi-core CPUs. Even single core processors can run multiple threads, with the OS determining (scheduling) which thread can run at which points. That means, it’s been around for a long time. And in that time we’ve developed a large number of ways that both help synchronise threads, and better isolate threads.</p>
<p>Whilst there’s no silver bullet, it’s my belief that a game engine should provide much better mechanics for performing asynchronous operations, actively encouraging splitting apart logic from the game loop. Unity goes so far as to provide a <a href="https://docs.unity3d.com/Manual/Coroutines.html">coroutine</a> mechanism which is a poor man’s version of the <code class="highlighter-rouge">async/await</code> functionality discussed earlier, with the added downside of not being intrinsically compatible with the .Net API surface.</p>
<p>We’ll call this problem number four.</p>
<h2 id="first-world-problems">First World Problems</h2>
<p>These are the problems I encountered on day 1. Unfortunately, it took much more than that first day to address each of these issues.</p>
<p><img src="../images/first-world-problems-01.jpg" alt="cookie too big" /></p>
<p>However, given this article has already hit over 2000 words, I’ll break out write-ups on how I went about solving these problems into a <a href="http://blog.devbot.net/tag/game-loop/">mini-series</a>. Subscribe or check back for solutions on each of the problems listed above!</p>Quite some time ago, following a short and shallow review of the game development industry, I started writing a game; Clean Space. Work on the game is ongoing and I have learned an incredible amount so far. Whilst I’ll happily talk the ears off of anyone in proximity that shows but a hint of interest, I’ve done very poorly at writing down some of the lessons I’ve learned throughout the course of Clean Space’s development. So, in this article, I want to talk about “The Game Loop”, how it fits into major Game Engines, and why I very quickly came to the conclusion that it is wrong.Brownfield ASP.Net Web Forms Migrations2017-12-13T00:00:00+00:002017-12-13T00:00:00+00:00http://blog.devbot.net/webforms-migrations<p>I recently started work for a <a href="http://www.futuredigital.co.uk/">new (and very exciting) company</a>. Like many of my previous roles, this one entails a good mix of <a href="http://en.wikipedia.org/wiki/Brownfield_(software_development)"><em>Brownfield</em></a> and <a href="http://en.wikipedia.org/wiki/Greenfield_project"><em>Greenfield</em></a> development. I’ve met and worked with some great developers who would outright refuse roles that held too much Brownfield work, preferring pastures green, as well as working with some great devs that prefer the challenge of Brownfield.</p>
<p><img src="../images/green-brown-field.png" alt="brown & green field" /></p>
<p style="font-size: 12px; text-align: center;"><em>Image from <a href="http://virtualgeek.typepad.com/virtual_geek/2015/08/vmworld-2015-cloud-native-apps-a-complex-interesting-but-fascinating-story.html">Virtual Geek</a></em></p>
<p>Having worked with the key <em>.Net</em> web technologies, I’d like to put to paper and potentially pass on, by way of this article, some insights into Brownfield development (plus invite comment on further insights and improvements on these techniques).</p>
<!-- markdownlint-disable MD007 -->
<!-- markdownlint-disable MD010 -->
<!-- TOC -->
<ul>
<li><a href="#estate">Estate</a>
<ul>
<li><a href="#tact">Tact</a></li>
<li><a href="#mining">Mining</a></li>
</ul>
</li>
<li><a href="#goal">Goal</a>
<ul>
<li><a href="#top-down">Top-Down</a></li>
<li><a href="#bottom-up">Bottom-Up</a></li>
</ul>
</li>
<li><a href="#migration">Migration</a>
<ul>
<li><a href="#asp-classic">ASP Classic</a></li>
<li><a href="#web-forms">Web Forms</a></li>
<li><a href="#mvc">MVC</a></li>
</ul>
</li>
<li><a href="#techniques">Techniques</a>
<ul>
<li><a href="#sideloading">Sideloading</a>
<ul>
<li><a href="#web-forms-to-mvc-5">Web Forms to MVC 5</a>
<ul>
<li><a href="#move-web-forms-into-mvc-5">Move Web Forms into MVC 5</a></li>
<li><a href="#move-mvc-5-into-web-forms">Move MVC 5 into Web Forms</a></li>
</ul>
</li>
<li><a href="#full-fat-net-core">Full-Fat .Net Core</a></li>
</ul>
</li>
<li><a href="#proxy">Proxy</a></li>
</ul>
</li>
<li><a href="#testing">Testing</a>
<ul>
<li><a href="#constructor-injection">Constructor Injection</a></li>
<li><a href="#property-injection">Property Injection</a></li>
</ul>
</li>
</ul>
<!-- /TOC -->
<!-- markdownlint-enable MD007 -->
<!-- markdownlint-enable MD010 -->
<h2 id="estate">Estate</h2>
<p>My past few roles have entailed a large amount of what we’ll fondly refer to as <em>heritage</em> software. Whenever tackling these code bases, my first task towards orchestrating the migration or uplifting of heritage software is establishing the <em>estate</em>.</p>
<p>Most of the legacy code I’ve inherited throughout my career was being maintained by developers that were not around for the entire history of the software. Oftentimes you’ll be working with developers long since devoid of the original authors and left with little more than speculation and grey area, but your number one job is to understand the functionality of the software suite.</p>
<p>I don’t aim to know how it works, and as amusing or aggravating as the <em>why-it-works-that-way</em> might be, that isn’t necessarily of importance either.</p>
<h3 id="tact">Tact</h3>
<p>Those that know me well probably wouldn’t use the word “tact” in the same sentence as my name. Bad code <strong>really</strong> gripes me and it’s not uncommon for my mouth to move before my brain catches up. In closed company this can be a great stress relief, and I’ve no problem with anonymous/redacted submissions to <a href="https://thedailywtf.com">The Daily WTF</a>, but putting people’s backs up doesn’t help.</p>
<p><img src="../images/your-code-is-bad.jpg" alt="you-code-is-bad" /></p>
<p style="font-size: 12px; text-align: center;"><em>Image from <a href="http://www.quickmeme.com/">Quick Meme</a></em></p>
<p>Having hurt several people’s feelings in the past saying the first angry thing that came to mind after seeing a snippet of… <em>interesting</em> code, I’ve had to lay some ground rules for myself that others may benefit from.</p>
<ul>
<li>Presume that the bizarre and downright crazy code you come across was driven by one reason or another that you simply can’t comprehend, and the inability to comprehend it is <em>your</em> shortcoming.</li>
<li>Accept that decisions were made with constraints you can’t fathom, and there’s little that whining about it will do to change a thing.</li>
</ul>
<p>I’m a big fan of of this quote</p>
<blockquote>
<p><em>“Always code as if the person who ends up maintaining your code is a violent psychopath who knows where you live.” - <a href="https://stackoverflow.com/questions/876089/who-wrote-this-programing-saying-always-code-as-if-the-guy-who-ends-up-maintai">John Wood (probably?)</a></em></p>
</blockquote>
<p>…but in all likelihood, the developers who wrote the software didn’t hear that advice.</p>
<p>There are only two real ways I’ve found to establish an estate. The first and most preferable is through the experience of others. Asking questions starting at high level design, digging down as necessary, but making an effort not to point fingers. If the people answering my questions are not aware how bad (I think) their code is, I’d rather teach them by example; through excited demonstrations of good software at a later date, <strong>not</strong> by shaming them in the present.</p>
<h3 id="mining">Mining</h3>
<p>Alongside gaining information from peers that may have been around longer than me, or in the complete absence of such, I also dig through code. Of course, the first thing I do is go looking for an up to date and comprehensive test suite… one day that might even pan out for me. More likely though, you’ll jump to a test project to find empty, redundant, commented-out, superfluous blocks of code.</p>
<p>My preferred approach when I don’t have tests is to establish the public endpoints. What is the customer facing surface area of the product? What are the common interactions? How does the code accomplish these tasks? What services, API calls, database logic and so forth is required to achieve the most basic and most common aspects?</p>
<p>I don’t try to memorise and graph everything; take notes and update any documentation that proved to be incorrect, but at this stage you just need to get a feel for how the thing hangs together.</p>
<h2 id="goal">Goal</h2>
<p>You might think that this section should have come earlier. But actually, the establishing of one’s estate is something I’d begin on day 1 of any role, regardless of overarching objectives.</p>
<p>I’m not sure if we’re seeing a shift in the industry or if I’ve just been lucky, but my last several CTO’s/managers valued intelligent, self-managing autonomous teams. They provided high level goals and appropriate deadlines with little else, preferring that the team determine the best courses of action.</p>
<p>If your managers aren’t quite as cool though, there’s still some tips below, especially in the <a href="#bottom-up">Bottom-Up Section</a>.</p>
<h3 id="top-down">Top-Down</h3>
<blockquote>
<p>Note: <em>I’ve noticed that I don’t always agree with which way round “top-down” and “bottom-up” describes these mechanisms. I’ll describe them as I visualise them in my head, but be aware I (or others) may use the inverse!</em></p>
</blockquote>
<p>Knowing what the product does, having established the estate, albeit not including all the nuances and finer detail, how would I write the product from scratch? As a team, draft and design, at a high level, what that system would look like. What tech stacks would you be looking at, what architecture, what deployment infrastructure, security, resilience, and so forth, would be built in?</p>
<p>This <em>ideal</em> system, subject to change as requirements are deduced and refined, is now my goal. The question becomes, how do you <em>iteratively</em> arrive at that goal.</p>
<p>In the past I’ve worked for a couple of companies that had decided that the only way to migrate their product, was a complete re-write. At the time, this meant more interesting code writing for me, so who was I to complain. However, one company was writing a new web product in parallel with maintenance on a thick WinForms product. I don’t know if they ever got there, but I had the distinct impression that there was simply no way for the new product to catch up with the old.</p>
<p>Another company I worked for had been through THREE attempts to re-write their software. Each had ended when cash or motivation for the cycle ran dry and the re-write was scrapped in favour of the much more stable and feature-full original legacy software. By the time I joined said company, they had bought all of their users a second monitor so that everyone could run new and old systems at the same time - entering data into both systems for every operation.</p>
<p><img src="https://vignette.wikia.nocookie.net/uncyclopedia/images/2/24/Funny_wtf_cat.jpg/revision/latest?cb=20090211025621" alt="wtf" /></p>
<p><em>Image from Uncyclopedia Wiki</em> {: style=”font-size: 12px; text-align: center;”}</p>
<p>The point being, if your plan is to <em>File -> New Solution</em> and one day flick a switch, chances are you’re doomed to failure. I’ve heard in passing from proud CTO’s where such an endeavour has paid off, but it is a massive and <em>unnecessary</em> gamble. By and large, all you’ll achieve is the creation of an incomplete product fork that is shelved when the cash runs out, or is forced into the wilderness despite reduced and unfinished functionality, causing your users no end of pain.</p>
<p>Therefore, the trick becomes, how do you get from A → B. I try to define phases, often utilising one of the <a href="#techniques">techniques below</a> as a starting point, then iteratively adding my <em>ideal</em> infrastructure in phases that get me from the current code base to the end goal. Whilst that’s quite an abstract description of this process, each system will need to be assessed on a case by case basis.</p>
<p>Some objectives I try to maintain whilst defining these phases may be of help:</p>
<ul>
<li>Avoid downtime
<ul>
<li>Especially when using the <a href="#proxy">proxy technique</a> described later in this article, it’s quite often possible to roll over to new infrastructure at the speed a DNS change propagates without incurring any downtime at all. In the case of upgrades to necessitate <a href="#sideloading">sideloading</a> however, the best bet is often to coincide changes with a scheduled release.</li>
</ul>
</li>
<li>Improve visibility
<ul>
<li>Along every step of the way, I try to improve the logs and analytics available to me. I know that speculation and even <em>educated</em> guesswork is still exactly that, guesswork.</li>
</ul>
</li>
<li>Maintain reverse compatibility
<ul>
<li>As heritage components are exchanged out, I do what I can to accommodate existing users. Legacy API surfaces typically don’t have any <a href="https://www.hanselman.com/blog/ASPNETCoreRESTfulWebAPIVersioningMadeEasy.aspx">versioning mechanisms</a> so treat your existing estate as <code class="highlighter-rouge">v1</code> implicitly, and build <code class="highlighter-rouge">v2</code> upon it, using metrics and logs to determine when aspects of the heritage system can safely be disabled and removed.</li>
</ul>
</li>
<li>Clean up orphaned work
<ul>
<li>As components are deprecated and replaced, it’s important to remember to go back and delete the respective code. Many a time I’ve stumbled across orphaned code but lacking the context that the functionality was recently replaced, the decision to remove the code is much trickier!</li>
</ul>
</li>
</ul>
<h3 id="bottom-up">Bottom-Up</h3>
<p>I’ve worked for several companies where terms like <em>“migration”</em> and <em>“re-write”</em> were too big and scary to be even uttered. Companies where product owners, line management, top bods, or all of the above were against the idea of improving the software for one reason or another. Sometimes it’s worth fighting with them over it; perhaps it’s just a fear of the unknown which can be alleviated, or a misunderstanding of what’s involved.</p>
<p>But when all else fails, I always fall back on the <em>bottom-up</em> approach. For those familiar with the <a href="https://books.google.co.uk/books?id=_i6bDeoCQzsC&pg=PA14&lpg=PA14#v=onepage&q&f=false">boyscout rule</a>, the bottom-up migration is little more than a slight extension of that. More often than not, I’ve found it’s just about drawing lines a little further in the sand.</p>
<p>I’ve often found that these roles follow <em>Panic</em>-Driven-Development (PDD), placing themselves in an endless vicious cycle of fire fighting the latest problem as quickly as possible, foregoing clean coding principles. Many a time I’ve seen this equally-endlessly described as a <em>temporary</em> measure, but of course, foregoing good practices in code inevitably leading to more fires to that need putting out, increasing code complexity from hack after hack further slowing down the ability to maintain the software.</p>
<p>The way out of this loop, I’ve found, is to move the line in the sand. Step up your boy-scouting from removing unused variables and correcting member casing, to writing tests. Using one of the <a href="#testing">testing practices</a> explained below, I’m often able to get dependency injection and mocking into even the worst of code bases.</p>
<p>Once you’ve established a testing practice (remembering to onboard and help peers to do the same), step it up to the next level. Keep improving upon the worst areas, driven by customer demand typically, though evaluation of technical debt where time permits. With this approach, I find you end up in a <em>hybrid</em> state for longer than is the case with <em>top-down</em>, but if your focus is on addressing problem areas, presumably any lingering legacy code isn’t causing you a great deal of harm.</p>
<p>In that respect, this approach, while not my preference, can be more palatable to stakeholders. Better yet, you don’t need any <em>permission</em> or <em>sign-off</em>.</p>
<p>Instead of saying to my product owner</p>
<blockquote>
<p><em>“If I do it <strong>properly</strong> it’ll take this long, but I can hack it in much quicker if needs be</em></p>
</blockquote>
<p>…I only offer them estimates based on my line in the sand. I’ve found there needs to be a pragmatic vs. dogmatic balance mind you. If you leave your product owner or stakeholders in a bind, you may be doing more harm than good to your company, no matter your intentions. More often than not there’s a middle ground that both sides can compromise and agree upon, though try not to be pushed too far back from that mythical line in the sand.</p>
<p>I’ve usually found this to be an arduous and lengthy process, but as confidence in the software increases due to a reducing number of bugs, confidence in the development team increases alongside it. If the doors were closed on a top-down approach before, you may find them increasingly ajar as confidence improves.</p>
<h2 id="migration">Migration</h2>
<p>Having established the estate, determined a preferred migration methodology, and put together some flexible high level phases that will get us to our goal, the really tough part begins. How to get there?</p>
<p>I’m going to list a few titbits I’ve picked up over the years, including code samples, where applicable, that may help you achieve your ends. These particular code samples will be focused on the web stack, but I’ve had experience with WinForms, WPF, UWP (desktop based) technologies so if you’d like some advice there, ping me a comment and I’ll draft up some notes.</p>
<p>If you have something specific in mind, feel free to jump to the section below that best describes where you are right now.</p>
<h3 id="asp-classic">ASP Classic</h3>
<p>I haven’t had to do this for a long while, but traditionally, facts are we’re running outside of .Net, and almost certainly on an outdated version of IIS and/or on bare metal. Options here really are very limited and dependent on circumstances, but my advice would be:</p>
<ul>
<li>If there is an IIS version that can side load both the ASP Classic site(s) and a .Net site, I’ll attempt to follow the <a href="#proxy">Proxy</a> section below.</li>
<li>If the site(s) are addressed by DNS instead of by IP directly, I can still try the <a href="#proxy">Proxy</a> methodology.</li>
<li>If the site is addressed directly, and/or the site relies on some crazy old undocumented and unsupported <code class="highlighter-rouge">dll</code>, inevitably I’ve ended up having to replace some components en masse. However, still check out the <a href="#testing">Testing</a> sections below for some advice on how to do so more safely.</li>
</ul>
<h3 id="web-forms">Web Forms</h3>
<p>As a technology, I consider <em>ASP .Net Web Forms</em> to be worse than ASP Classic. The bastardisation of HTTP to better suit desktop minded developers, whilst a viable <em>developer</em> migration strategy for Microsoft, was in all other respects a mistake. ASP Classic is at least not all that different from <em>“modern”</em> scripting languages such as PHP, which whilst I might not have any fondness for either, still have their uses.</p>
<p>I recommend checking out existing articles such as <a href="http://rachelappel.com/integrating-aspnet-web-forms-and-aspnetmvc/">Rachel Appel’s guide</a> or check out my own advice in the <a href="#web-forms-to-mvc-5">WebForms to MVC section below</a> for a guide on sideloading Web Forms and MVC. This allows the introduction of MVC mechanisms into an existing web application.</p>
<p>Once the infrastructure is in place, I make it a rule that all new work should be completed using the MVC architecture, and whenever a significant amount of work is required within an existing Web Form, I consider moving the code to MVC instead (bearing in mind aforementioned pragmatic/dogmatic caveat).</p>
<p>I’m not going to argue or even make a case for the merits of migrating from WebForms to MVC; I’d just end up ranting. Apart from my deepfelt resentment for the WebForms stack, the benefits of MVC over Web Forms are very easy to google for.</p>
<p>Once MVC side loading is in place, check out the <a href="#testing">Testing</a> section below for tips on how to more safely migrate heritage code.</p>
<h3 id="mvc">MVC</h3>
<p>I’ve been working with the <em>.Net Core</em> stack since it was still called <em>“Project K”</em> (so, a few years now). Whilst I’m more inclined to harp on about why and exactly how <em>.Net Core</em> is better, I’ll try not to. However, I do consider it a <em>huge</em> improvement over class/full-fat MVC (let alone Web Forms), and recommend it to others.</p>
<p>If you check out the section below on <a href="#full-fat-net-core">Full-Fat .Net Core</a>, you may find it’s easier to get into than you imagined.</p>
<h2 id="techniques">Techniques</h2>
<p>There are two key techniques I use and recommend to others when it comes to migrating a code base. They each have their own pros & cons, and the best choice will be determined by your circumstances.</p>
<p>All techniques assume you’re using some kind of version control and if everything goes wrong, you can just back out and start over. If you’re not using version control and mess things up, I’ll lose no sleep over your lack of forethought.</p>
<h3 id="sideloading">Sideloading</h3>
<p>This involves running both old and new technology stacks in the same application. It won’t apply to <em>ASP Classic</em>, but for later technologies, check out the following sections for examples on how to do this.</p>
<h4 id="web-forms-to-mvc-5">Web Forms to MVC 5</h4>
<p>There are already some great guides out there on how to sideload so I won’t provide all the details here, they haven’t really changed that much between the MVC versions. It’s important to note that this is a migration to the <em>“full fat”</em> ASP .Net framework and has nothing at all to do with <em>.Net Core</em>.</p>
<p>There are two ways to perform the migration as detailed below. Neither is necessarily <em>better</em>, but if you find yourself stuck trying one option, simply shelve/revert your changes and try the other option.</p>
<p>Regardless which option you go with, you’ll need to ensure your Web Forms application is targeting a supported version of <em>.Net</em>.</p>
<blockquote>
<p>Note: If you’re still targeting <em>.Net 4.5.1</em> or lower, you’re using an <a href="https://blogs.msdn.microsoft.com/dotnet/2015/12/09/support-ending-for-the-net-framework-4-4-5-and-4-5-1/">unsupported <em>.Net</em> version</a>, so get yourself updated.</p>
</blockquote>
<p>These infrastructural changes can be a nightmare to test if you don’t have a reliable test suite. There is no <em>silver bullet</em> if you don’t have tests; you’re going to need to set yourself up a replica of your Production server and make sure the upgraded version performs as expected the hard way (manual testing). For what it’s worth, whilst I have encountered issues post-migration, they’ve always been immediately obvious (errors on startup) and easy to google a resolution to. It’ll be tedious, but if your google-foo is strong, rarely more than a day or two of effort at least getting everything up and running.</p>
<h5 id="move-web-forms-into-mvc-5">Move Web Forms into MVC 5</h5>
<p>In this approach, we’ll create a new MVC 5 project and move our existing code over.</p>
<ol>
<li>Open one of the code files in the Web Forms project (doesn’t matter which).</li>
<li>Add a random (valid, but unused) <code class="highlighter-rouge">using</code> statement (unless there are already greyed out <code class="highlighter-rouge">using</code> statements present).
<ul>
<li>For example, if it’s not present already, add <code class="highlighter-rouge">using System.Web.Globalization;</code></li>
</ul>
</li>
<li>
<p>Now, with the caret on the unused <code class="highlighter-rouge">using</code> (which should be showing as greyed out if you’re using a modern Visual Studio), press <code class="highlighter-rouge">Ctrl + . (period)</code>, mouse over <code class="highlighter-rouge">Remove Unnecessary Usings</code> and then click <code class="highlighter-rouge">Solution</code>:</p>
<p><img src="../images/unused-usings.png" alt="unnecessary usings" /></p>
<p>This will come in very useful later if you haven’t been cleaning up your <code class="highlighter-rouge">using</code>s before now.</p>
<blockquote>
<p>Caveat: Visual Studio can be a little <em>cavalier</em> when determining whether a <code class="highlighter-rouge">using</code> is in use or not. Especially if you have any imports inside <code class="highlighter-rouge">aspx</code>/<code class="highlighter-rouge">asmx</code>/<code class="highlighter-rouge">ascx</code> files. Make sure you review your changes, that your project still builds, and that your views still render. Typically, opening any file modified will cause static analysis to kick in - if it starts showing exceptions, then add the necessary imports back.</p>
</blockquote>
</li>
<li>Create a new MVC 5 Project.</li>
<li>
<p>When prompted, ensure you enable Web Forms and MVC (and Web API if you will need it):</p>
<p><img src="../images/new-mvc5-app.png" alt="new mvc application" /></p>
</li>
<li>Navigate to your Web Forms project in Windows Explorer, and copy all of the files (and folders) to your New Project, except those listed below. If prompted, skip any files about replacements and take note of which files couldn’t be copied.
<ul>
<li><code class="highlighter-rouge">bin\</code></li>
<li><code class="highlighter-rouge">obj\</code></li>
<li><code class="highlighter-rouge">Properties\</code></li>
<li><code class="highlighter-rouge">Connect Services\</code></li>
<li><code class="highlighter-rouge">*.csproj</code></li>
<li><code class="highlighter-rouge">global.asax</code></li>
<li><code class="highlighter-rouge">global.asax.cs</code></li>
<li><code class="highlighter-rouge">web.config</code></li>
</ul>
</li>
<li>
<p>In Visual Studio, select your MVC project in <em>Solution Explorer</em> and select <em>Show All Files</em>:</p>
<p><img src="../images/show-all-files.png" alt="show all files" /></p>
</li>
<li>
<p>If you expand your MVC project, in addition to the default MVC infrastructure generated during project creation, you should be able to see all your pages, controls, and so forth, copied over from the Web Forms project. You need to work through all these files, R-Click them and choose <code class="highlighter-rouge">Include in Project</code>. You can highlight several at a time and include whole folders to make it easier.</p>
<p><img src="../images/include-in-project.png" alt="include in project" /></p>
</li>
<li>Merge the Web Forms <code class="highlighter-rouge">global.asax</code> into your new MVC 5 <code class="highlighter-rouge">global.asax</code>
<ul>
<li>This should be no different to any other code merge. MVC 5 will only be using the <code class="highlighter-rouge">Application_Start</code> event currently, so copy over any custom code you have in the old <code class="highlighter-rouge">global.asax</code> being careful not to remove the MVC 5 logic.</li>
</ul>
</li>
<li>Merge the Web Forms <code class="highlighter-rouge">web.config</code> into your new MVC 5 <code class="highlighter-rouge">web.config</code>
<ul>
<li>This is much trickier because there will often be a considerable difference between the configs. You’ll need to not only copy over the obvious items like <code class="highlighter-rouge">appSettings</code> and <code class="highlighter-rouge">connectionStrings</code>, but also ensure required <code class="highlighter-rouge">HttpModules</code>, security and runtime settings, WCF bindings, and anything else your Web Forms project <em>actually</em> needed is copied over. If you start with the basics, you can then just keep trying to load your new site, resolving startup errors by copying over the respective old configuration (you may need to come back to this task after you’ve got the new project compiling).</li>
</ul>
</li>
<li>Merge over all the files you took note of in <em>Step 6</em>.</li>
<li>Try to build your new MVC project. The likelihood is that it will fail because references included in your old project have not yet been added to the MVC project.
<ul>
<li>Run a build and then examine your <code class="highlighter-rouge">Error List</code>, ensuring that you’ve selected <code class="highlighter-rouge">Build Only</code> in the drop down (Intellisense can generate thousands of errors when a reference is missing, creating a lot of confusing noise).</li>
<li>The errors you find will almost always be a difference in references between old and new projects. In the file with an exception, take a look at what Visual Studio thinks is an unused or missing <code class="highlighter-rouge">using</code> statement (which we know shouldn’t exist now because we removed them all earlier) at the top of the file, and add a reference to the respective assembly. The name of the <code class="highlighter-rouge">using</code> should give a good indication as to which assembly you’re missing, otherwise lookup one of the classes that cannot be resolved on MSDN; the page will tell you exactly which assembly contains the given class.</li>
<li>Any errors you can’t fix by adding a reference to the necessary assembly, take to google.</li>
</ul>
</li>
<li>Make sure your views actually compile. Either open each of the views to see what intellisense finds, or publish the site with <a href="https://docs.microsoft.com/en-us/aspnet/web-forms/overview/older-versions-getting-started/deploying-web-site-projects/precompiling-your-website-cs">Precompilation enabled</a> to ensure the views haven’t been broken.
<ul>
<li>For whatever reason, the static analysis that identifies problems in your <code class="highlighter-rouge">*.cs</code> files doesn’t scan your views so this form of precompilation can be useful to locate errors that static analysis misses and would otherwise cause you a runtime exception.</li>
</ul>
</li>
</ol>
<p>Congratulations, with a little bit of luck you’ve now got your project sideloading. Feel free to drop me a comment if you get stuck and can’t find an answer on google.</p>
<p>Next steps, you can either look at the <a href="#testing">Testing</a> section for advice on how to replace those pesky Web Forms files, or if you’ve managed to achieve that, check out the section below on <a href="#full-fat-net-core">migrating to .Net core</a>.</p>
<h5 id="move-mvc-5-into-web-forms">Move MVC 5 into Web Forms</h5>
<p>With this approach, we’ll try to uplift an existing Web Forms project to include the necessary MVC 5 infrastructure. Before you start, I recommend creating a new (temporary) MVC 5 web application that has both Web Forms and MVC enabled so that you have an easy-to-reach example project to check if you’re unsure about something.</p>
<ol>
<li>Install the <a href="https://www.nuget.org/packages/microsoft.aspnet.mvc"><code class="highlighter-rouge">Microsoft.AspNet.Mvc</code> NuGet Package</a></li>
<li>
<p>Add the following <code class="highlighter-rouge">configuration/appSettings</code> in <code class="highlighter-rouge">web.config</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
</pre></td><td class="rouge-code"><pre> <span class="nt"><appSettings></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"webpages:Version"</span> <span class="na">value=</span><span class="s">"3.0.0.0"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"webpages:Enabled"</span> <span class="na">value=</span><span class="s">"false"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"PreserveLoginUrl"</span> <span class="na">value=</span><span class="s">"true"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"ClientValidationEnabled"</span> <span class="na">value=</span><span class="s">"true"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"UnobtrusiveJavaScriptEnabled"</span> <span class="na">value=</span><span class="s">"true"</span><span class="nt">/></span>
<span class="nt"></appSettings></span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Add the following section under <code class="highlighter-rouge">configuration/system.web</code> in <code class="highlighter-rouge">web.config</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre> <span class="nt"><pages></span>
<span class="nt"><namespaces></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Helpers"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc.Ajax"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc.Html"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Routing"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.WebPages"</span><span class="nt">/></span>
<span class="nt"></namespaces></span>
<span class="nt"></pages></span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>Add a directory called <code class="highlighter-rouge">App_Start</code></li>
<li>
<p>Add a class called <code class="highlighter-rouge">RouteConfig</code> to the <code class="highlighter-rouge">App_Start</code> directory, with the following content (update the Namespace accordingly):</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</pre></td><td class="rouge-code"><pre> <span class="k">using</span> <span class="nn">System.Web.Mvc</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web.Routing</span><span class="p">;</span>
<span class="k">namespace</span> <span class="err">$</span><span class="nn">Namespace</span><span class="err">$</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">RouteConfig</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">void</span> <span class="nf">RegisterRoutes</span><span class="p">(</span><span class="n">RouteCollection</span> <span class="n">routes</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">routes</span><span class="p">.</span><span class="nf">IgnoreRoute</span><span class="p">(</span><span class="s">"{resource}.axd/{*pathInfo}"</span><span class="p">);</span>
<span class="n">routes</span><span class="p">.</span><span class="nf">MapRoute</span><span class="p">(</span>
<span class="n">name</span><span class="p">:</span> <span class="s">"Default"</span><span class="p">,</span>
<span class="n">url</span><span class="p">:</span> <span class="s">"{controller}/{action}/{id}"</span><span class="p">,</span>
<span class="n">defaults</span><span class="p">:</span> <span class="k">new</span> <span class="p">{</span> <span class="n">controller</span> <span class="p">=</span> <span class="s">"Home"</span><span class="p">,</span> <span class="n">action</span> <span class="p">=</span> <span class="s">"Index"</span><span class="p">,</span> <span class="n">id</span> <span class="p">=</span> <span class="n">UrlParameter</span><span class="p">.</span><span class="n">Optional</span> <span class="p">}</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>Add the following lines to the <code class="highlighter-rouge">Application_Start</code> event in <code class="highlighter-rouge">global.asax</code>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
</pre></td><td class="rouge-code"><pre> <span class="n">AreaRegistration</span><span class="p">.</span><span class="nf">RegisterAllAreas</span><span class="p">();</span>
<span class="n">RouteConfig</span><span class="p">.</span><span class="nf">RegisterRoutes</span><span class="p">(</span><span class="n">RouteTable</span><span class="p">.</span><span class="n">Routes</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<blockquote>
<p>Note: You’ll need the following <code class="highlighter-rouge">using</code> statements if not present:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web.Mvc</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web.Routing</span><span class="p">;</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</blockquote>
</li>
<li>Add the following directories to the solution:
<ul>
<li><code class="highlighter-rouge">Controllers</code></li>
<li><code class="highlighter-rouge">Models</code></li>
<li><code class="highlighter-rouge">Views</code></li>
</ul>
</li>
<li>
<p>Add a <em>second</em> <code class="highlighter-rouge">web.config</code> to the <code class="highlighter-rouge">Views</code> directory, with the following content:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
</pre></td><td class="rouge-code"><pre> <span class="cp"><?xml version="1.0"?></span>
<span class="nt"><configuration></span>
<span class="nt"><configSections></span>
<span class="nt"><sectionGroup</span> <span class="na">name=</span><span class="s">"system.web.webPages.razor"</span> <span class="na">type=</span><span class="s">"System.Web.WebPages.Razor.Configuration.RazorWebSectionGroup, System.Web.WebPages.Razor, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"</span><span class="nt">></span>
<span class="nt"><section</span> <span class="na">name=</span><span class="s">"host"</span> <span class="na">type=</span><span class="s">"System.Web.WebPages.Razor.Configuration.HostSection, System.Web.WebPages.Razor, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"</span> <span class="na">requirePermission=</span><span class="s">"false"</span> <span class="nt">/></span>
<span class="nt"><section</span> <span class="na">name=</span><span class="s">"pages"</span> <span class="na">type=</span><span class="s">"System.Web.WebPages.Razor.Configuration.RazorPagesSection, System.Web.WebPages.Razor, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"</span> <span class="na">requirePermission=</span><span class="s">"false"</span> <span class="nt">/></span>
<span class="nt"></sectionGroup></span>
<span class="nt"></configSections></span>
<span class="nt"><system.web.webPages.razor></span>
<span class="nt"><host</span> <span class="na">factoryType=</span><span class="s">"System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"</span> <span class="nt">/></span>
<span class="nt"><pages</span> <span class="na">pageBaseType=</span><span class="s">"System.Web.Mvc.WebViewPage"</span><span class="nt">></span>
<span class="nt"><namespaces></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc"</span> <span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc.Ajax"</span> <span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Mvc.Html"</span> <span class="nt">/></span>
<span class="nt"><add</span> <span class="na">namespace=</span><span class="s">"System.Web.Routing"</span> <span class="nt">/></span>
<span class="nt"></namespaces></span>
<span class="nt"></pages></span>
<span class="nt"></system.web.webPages.razor></span>
<span class="nt"><appSettings></span>
<span class="nt"><add</span> <span class="na">key=</span><span class="s">"webpages:Enabled"</span> <span class="na">value=</span><span class="s">"false"</span> <span class="nt">/></span>
<span class="nt"></appSettings></span>
<span class="nt"><system.webServer></span>
<span class="nt"><handlers></span>
<span class="nt"><remove</span> <span class="na">name=</span><span class="s">"BlockViewHandler"</span><span class="nt">/></span>
<span class="nt"><add</span> <span class="na">name=</span><span class="s">"BlockViewHandler"</span> <span class="na">path=</span><span class="s">"*"</span> <span class="na">verb=</span><span class="s">"*"</span> <span class="na">preCondition=</span><span class="s">"integratedMode"</span> <span class="na">type=</span><span class="s">"System.Web.HttpNotFoundHandler"</span> <span class="nt">/></span>
<span class="nt"></handlers></span>
<span class="nt"></system.webServer></span>
<span class="nt"><system.web></span>
<span class="nt"><compilation></span>
<span class="nt"><assemblies></span>
<span class="nt"><add</span> <span class="na">assembly=</span><span class="s">"System.Web.Mvc, Version=5.2.3.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35"</span> <span class="nt">/></span>
<span class="nt"></assemblies></span>
<span class="nt"></compilation></span>
<span class="nt"></system.web></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>In this same <em>second</em> <code class="highlighter-rouge">web.config</code>, add your own namespace next to the <code class="highlighter-rouge">System.Web</code> namespaces under the <code class="highlighter-rouge">configuration/system.web.webPages.razor/pages/namespaces</code> section
<ul>
<li>For example, <code class="highlighter-rouge"><add namespace="Your.Namespace" /></code></li>
</ul>
</li>
</ol>
<p>All being well, you should now have enabled MVC in your Web Forms project. To test everything is working, try <a href="https://docs.microsoft.com/en-us/aspnet/mvc/overview/getting-started/introduction/adding-a-controller">Adding a Controller</a> and confirming you get a result.</p>
<p>Next steps, you can either look at the <a href="#testing">Testing</a> section for advice on how to replace those pesky Web Forms files, or if you’ve managed to achieve that, check out the section below on <a href="#full-fat-net-core">migrating to .Net core</a>.</p>
<h4 id="full-fat-net-core">Full-Fat .Net Core</h4>
<p>For developers that haven’t jumped into <em>.Net Core</em> yet, I appreciate it can be a little daunting. A great deal has changed. You may have heard something along the lines of</p>
<blockquote>
<p><em>“.Net Core can run on the Full Framework”</em></p>
</blockquote>
<p>..but that probably doesn’t make a great deal of sense. I don’t really want to get into explaining the difference between platforms or what exactly <em>.Net Standard</em> is, so I recommend having a scan of <a href="https://docs.microsoft.com/en-us/dotnet/standard/net-standard">this article</a> if you’re unsure.</p>
<p>The point being, when people talk about <em>“Full Fat .Net Core”</em> what they’re actually talking about is using the latest tooling such as the improved <a href="https://docs.microsoft.com/en-us/dotnet/core/tools/csproj"><code class="highlighter-rouge">csproj</code> format</a> and <em>not</em> the cross-platform features introduced by <em>.Net Core</em> itself.</p>
<p>With that cleared up, you can very easily begin using the new tooling by targeting your new style <code class="highlighter-rouge">csproj</code> at the full <em>.Net Framework</em>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
</pre></td><td class="rouge-code"><pre><span class="nt"><Project</span> <span class="na">Sdk=</span><span class="s">"Microsoft.NET.Sdk.Web"</span><span class="nt">></span>
<span class="nt"><PropertyGroup></span>
<span class="nt"><TargetFramework></span>net462<span class="nt"></TargetFramework></span>
<span class="nt"></PropertyGroup></span>
<span class="nt"><ItemGroup></span>
<span class="nt"><Folder</span> <span class="na">Include=</span><span class="s">"wwwroot\"</span> <span class="nt">/></span>
<span class="nt"></ItemGroup></span>
<span class="nt"><ItemGroup></span>
<span class="nt"><PackageReference</span> <span class="na">Include=</span><span class="s">"Microsoft.AspNetCore.All"</span> <span class="na">Version=</span><span class="s">"2.0.3"</span> <span class="nt">/></span>
<span class="nt"></ItemGroup></span>
<span class="nt"></Project></span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>These new style <code class="highlighter-rouge">csproj</code> files have gone the way of old school <code class="highlighter-rouge">web.config</code> and had a bunch of stuff hidden away so that, hopefully, only content that matters to you should be present. No longer will you find all the default build configuration and individual entries for each file in your project. Instead, the build configuration has some defaults tucked away (which can be overridden). For example, by default:</p>
<ul>
<li>All <code class="highlighter-rouge">*.cs</code> files are included and compiled</li>
<li>All <code class="highlighter-rouge">*.resx</code> files are included and embedded</li>
<li>All other files are shown in the project, but have no impact on the build</li>
</ul>
<p>In addition to a cleaner <code class="highlighter-rouge">csproj</code>, you also get features like the <a href="https://www.erikheemskerk.nl/transitive-nuget-dependencies-net-core-got-your-back/"><em>transitive dependency</em></a> mechanism. When something is referenced, everything it references in turn is also pulled in, so instead of a <code class="highlighter-rouge">package.config</code> (which is no longer necessary) filled with dozens of packages despite you only having added 1 top-level package, you need only list the top level now.</p>
<p><em>.Net Core</em> has a much higher release cadence than <em>.Net Framework</em>, so the associated tooling evolves quicker; it’s likely worth your time <a href="https://docs.microsoft.com/en-us/aspnet/core/migration/">migrating to .Net Core</a>. You’ll be rewarded with improved tooling, and you needn’t change anything on your production server / deployment targets. Plus, it’ll place you in a better position to migrate to the <code class="highlighter-rouge">.Net Core</code> <em>platform</em> should it’s feature set entice you.</p>
<h3 id="proxy">Proxy</h3>
<p>If <a href="#sideloading">sideloading</a> isn’t for you, there is an alternative. The idea here is to create a new web application which completely replaces your old website, but, by default, forwards all the requests to your old site (which is deployed elsewhere, in turn).</p>
<p>This of course introduces an additional web call and the associated latency which can be a deal breaker for many. But, if old and new site are physically co-located or sat in the same Cloud virtual network, you’ll be surprised how small that latency is.</p>
<p>What you get in return for this latency is the ability to conditionally handle any request originally intended for your old site whilst using a new and improved technology stack, safe in the knowledge that anything you haven’t decided to replace is being executed the way it used to.</p>
<p>Better still, Microsoft have already started writing the code necessary to do this very simply.</p>
<blockquote>
<p>Caveat: I’m going to reference the <code class="highlighter-rouge">dev</code> branch of an unpublished Microsoft GitHub repository in the following snippets because MS haven’t officially published this anywhere yet - if that scares the snot out of you, you could always try writing it yourself… <em>/sigh</em></p>
<p>In my opinion, better to use the existing code with no expectation of support, pushing useful changes back and testing the middleware for Microsoft than to write something homebrew.</p>
</blockquote>
<p>I’m going to assume that you’re using Git and the new site you’re introducing is <em>.Net Core</em> (running on <a href="#full-fat-net-core">full fat</a> if need be) because why wouldn’t you? (Sure, because you want a mature EF, working SignalR, some other un-migrated dependency; all legit reasons).</p>
<ol>
<li>
<p>Create a new <em>“ASP .NET Core Web Application”</em></p>
<p><img src="../images/new-netcore-webapp.png" alt="new net core web app" /></p>
<ul>
<li>If you need to target <em>.Net Framework</em>, open the <code class="highlighter-rouge">csproj</code> and change the value of <code class="highlighter-rouge"><TargetFramework></code> (i.e. <code class="highlighter-rouge">net462</code> for .Net Framework v4.6.2).</li>
</ul>
</li>
<li><a href="https://github.com/aspnet/Proxy#fork-destination-box">Fork Microsoft’s Proxy repository</a> or feel free to use the one I <a href="https://github.com/futuredigitalfootprint/Proxy">created for Future Digital Footprint</a>
<ul>
<li>If you’re going to use ours, you can skip to step 9</li>
</ul>
</li>
<li><a href="https://git-scm.com/docs/git-submodule">Submodule</a> your fork</li>
<li>Switch your submodule to the <code class="highlighter-rouge">dev</code> branch</li>
<li>Update <code class="highlighter-rouge">Microsoft.AspNetCore.Proxy.csproj</code> to reference the latest <em>stable</em> (not nightly) versions of the <a href="https://www.nuget.org/packages/microsoft.aspnetcore.websockets"><code class="highlighter-rouge">Microsoft.AspNetCore.WebSockets</code></a> and <a href="https://www.nuget.org/packages/microsoft.extensions.options"><code class="highlighter-rouge">Microsoft.Extensions.Options</code></a> projects
<ul>
<li>For example, see <a href="https://github.com/futuredigitalfootprint/Proxy/commit/583e3419fc3d9cb363c9425de5069293fb766659#diff-87a66f67dbf25670909ed818b769a682">here</a></li>
</ul>
</li>
<li>Delete <code class="highlighter-rouge">src/Directory.Build.props</code> to bypass Microsoft’s build shenanigans</li>
<li>Commit and push the changes to your fork</li>
<li>Hop back into your main repository and commit the submodule change</li>
<li>In Visual Studio, R-Click the solution and <code class="highlighter-rouge">Add Existing Project...</code></li>
<li>Navigate to and select the submoduled <code class="highlighter-rouge">Microsoft.AspNetCore.Proxy.csproj</code></li>
<li>
<p>In your new web site, add a reference to the Proxy project:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
</pre></td><td class="rouge-code"><pre> <span class="nt"><Project</span> <span class="na">Sdk=</span><span class="s">"Microsoft.NET.Sdk.Web"</span><span class="nt">></span>
<span class="nt"><PropertyGroup></span>
<span class="nt"><TargetFramework></span>netcoreapp2.0<span class="nt"></TargetFramework></span>
<span class="nt"></PropertyGroup></span>
<span class="nt"><ItemGroup></span>
<span class="nt"><Folder</span> <span class="na">Include=</span><span class="s">"wwwroot\"</span> <span class="nt">/></span>
<span class="nt"></ItemGroup></span>
<span class="nt"><ItemGroup></span>
<span class="nt"><PackageReference</span> <span class="na">Include=</span><span class="s">"Microsoft.AspNetCore.All"</span> <span class="na">Version=</span><span class="s">"2.0.3"</span> <span class="nt">/></span>
<span class="nt"></ItemGroup></span>
<span class="nt"><ItemGroup></span>
<span class="nt"><ProjectReference</span> <span class="na">Include=</span><span class="s">"..\submodules\Microsoft.AspNetCore.Proxy\src\Microsoft.AspNetCore.Proxy\Microsoft.AspNetCore.Proxy.csproj"</span> <span class="nt">/></span>
<span class="nt"></ItemGroup></span>
<span class="nt"></Project></span>
</pre></td></tr></tbody></table></code></pre></div> </div>
<blockquote>
<p>Note: <em>Please don’t jump through all the above hoops if Microsoft have gotten round to publishing this package since I wrote this article. Be sure to check <a href="https://www.nuget.org/packages/microsoft.aspnetcore.proxy">here</a> for a version greater than <code class="highlighter-rouge">0.2.0</code>, at which point skip steps 2-11 and just add the NuGet package instead.</em></p>
</blockquote>
</li>
<li>
<p>Update your <code class="highlighter-rouge">Startup</code> class to add the Proxy:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
</pre></td><td class="rouge-code"><pre> <span class="k">using</span> <span class="nn">Microsoft.AspNetCore.Builder</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.AspNetCore.Hosting</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.AspNetCore.Http</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.AspNetCore.Proxy</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.Extensions.Configuration</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="c1">// your namespace</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Startup</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IConfiguration</span> <span class="n">_configuration</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">Startup</span><span class="p">(</span><span class="n">IConfiguration</span> <span class="n">configuration</span><span class="p">)</span>
<span class="p">=></span> <span class="n">_configuration</span> <span class="p">=</span> <span class="n">configuration</span><span class="p">;</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">ConfigureServices</span><span class="p">(</span><span class="n">IServiceCollection</span> <span class="n">services</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">services</span><span class="p">.</span><span class="nf">AddProxy</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Configure</span><span class="p">(</span><span class="n">IApplicationBuilder</span> <span class="n">app</span><span class="p">,</span> <span class="n">IHostingEnvironment</span> <span class="n">env</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">env</span><span class="p">.</span><span class="nf">IsDevelopment</span><span class="p">())</span>
<span class="n">app</span><span class="p">.</span><span class="nf">UseDeveloperExceptionPage</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">proxyOptions</span> <span class="p">=</span> <span class="n">_configuration</span><span class="p">.</span><span class="n">Get</span><span class="p"><</span><span class="n">ProxyOptions</span><span class="p">>();</span>
<span class="c1">// Host and AppendQuery properties will not bind correctly so we have to intervene</span>
<span class="n">proxyOptions</span><span class="p">.</span><span class="n">Host</span> <span class="p">=</span> <span class="n">HostString</span><span class="p">.</span><span class="nf">FromUriComponent</span><span class="p">(</span><span class="n">_configuration</span><span class="p">[</span><span class="s">"Host"</span><span class="p">]);</span>
<span class="n">proxyOptions</span><span class="p">.</span><span class="n">AppendQuery</span> <span class="p">=</span> <span class="n">QueryString</span><span class="p">.</span><span class="nf">FromUriComponent</span><span class="p">(</span><span class="n">_configuration</span><span class="p">[</span><span class="s">"AppendQuery"</span><span class="p">]);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">RunProxy</span><span class="p">(</span><span class="n">proxyOptions</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div> </div>
</li>
<li>
<p>You’ll also need to add details of where the old site will be available so that the proxy middleware knows where to forward requests to. You can use any <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?tabs=basicconfiguration">configuration style</a> you want, but as an example, here it is in <code class="highlighter-rouge">appSettings.json</code>:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Proxy"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Scheme"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https"</span><span class="p">,</span><span class="w">
</span><span class="nl">"Host"</span><span class="p">:</span><span class="w"> </span><span class="s2">"example.com"</span><span class="p">,</span><span class="w">
</span><span class="nl">"PathBase"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/virtualPath"</span><span class="p">,</span><span class="w">
</span><span class="nl">"AppendQuery"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foo=bar"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div> </div>
</li>
</ol>
<p>And that’s your proxy ready. All requests received by this web application will now be forwarded. But, it’s unlikely you’ll want to forward <em>all</em> requests. You can use normal routing mechanics so that, for example, a given request goes to some specific <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/middleware?tabs=aspnetcore2x">custom middleware</a>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">void</span> <span class="nf">Configure</span><span class="p">(</span><span class="n">IApplicationBuilder</span> <span class="n">app</span><span class="p">,</span> <span class="n">IHostingEnvironment</span> <span class="n">env</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">env</span><span class="p">.</span><span class="nf">IsDevelopment</span><span class="p">())</span>
<span class="n">app</span><span class="p">.</span><span class="nf">UseDeveloperExceptionPage</span><span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="nf">UseWhen</span><span class="p">(</span>
<span class="n">context</span> <span class="p">=></span> <span class="n">context</span><span class="p">.</span><span class="n">Request</span><span class="p">.</span><span class="n">Path</span><span class="p">.</span><span class="nf">StartsWithSegments</span><span class="p">(</span><span class="s">"/some-url"</span><span class="p">),</span>
<span class="n">builder</span> <span class="p">=></span> <span class="n">builder</span><span class="p">.</span><span class="n">UseMiddleware</span><span class="p"><</span><span class="n">MyCustomerMiddleware</span><span class="p">>());</span>
<span class="kt">var</span> <span class="n">proxyOptions</span> <span class="p">=</span> <span class="n">_configuration</span><span class="p">.</span><span class="n">Get</span><span class="p"><</span><span class="n">ProxyOptions</span><span class="p">>();</span>
<span class="c1">// Host and AppendQuery properties will not bind correctly so we have to intervene</span>
<span class="n">proxyOptions</span><span class="p">.</span><span class="n">Host</span> <span class="p">=</span> <span class="n">HostString</span><span class="p">.</span><span class="nf">FromUriComponent</span><span class="p">(</span><span class="n">_configuration</span><span class="p">[</span><span class="s">"Host"</span><span class="p">]);</span>
<span class="n">proxyOptions</span><span class="p">.</span><span class="n">AppendQuery</span> <span class="p">=</span> <span class="n">QueryString</span><span class="p">.</span><span class="nf">FromUriComponent</span><span class="p">(</span><span class="n">_configuration</span><span class="p">[</span><span class="s">"AppendQuery"</span><span class="p">]);</span>
<span class="n">app</span><span class="p">.</span><span class="nf">RunProxy</span><span class="p">(</span><span class="n">proxyOptions</span><span class="p">);</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Unfortunately, both the MVC and Proxy middleware are <em>pipeline branch terminators</em> which means that MVC will always handle a request and not pass it on to any middleware registered <em>after</em> the MVC registration (even when MVC doesn’t know how to handle it). And as you might imagine, there is no request the Proxy wouldn’t know how to handle because all it’s doing is forwarding requests to the configured endpoint, so again, it won’t pass a request on to the next piece of middleware. If you want to use MVC in your new site you’ll need to map specific requests to either the proxy <em>or</em> MVC (depending which way round you prefer / is easier).</p>
<h2 id="testing">Testing</h2>
<p>So you’ve decided which migration technique to use, be it <a href="#sideloading">sideloading</a> or via <a href="#proxy">proxy</a>, and now you’re ready to start replacing some of your old code? As I’ve said before in this article, there is no easy answer. However, I would like to show you the multi-targeting power of <em>.Net Core</em> tooling which may allow you to safely replace existing pages. Before we can do that though, you’re almost definitely going to need <a href="https://en.wikipedia.org/wiki/Dependency_injection"><em>dependency injection (DI)</em></a> in order to <a href="https://en.wikipedia.org/wiki/Mock_object"><em>mock</em></a> aspects of your environment and execution pipeline.</p>
<p>If you need to inject <code class="highlighter-rouge">*.aspx</code> pages, it’s a real palaver (one of the many benefits of moving to MVC is improved abstraction). There are only two mechanisms I know of for Web Forms described in the sections below.</p>
<h3 id="constructor-injection">Constructor Injection</h3>
<p>The reason you can’t easily implement DI in Web Forms is that the factories that are responsible for receiving a request and building up an instance of your Page to pass the request to, insist upon there being a parameterless public constructor on the page. If there can be no parameters, there can be no (constructor) injection.</p>
<p>You <em>could</em> circumvent this using the <a href="http://blog.ploeh.dk/2010/02/03/ServiceLocatorisanAnti-Pattern/">Service Locator Anti-Pattern</a>, but you <em>should not</em>. There are other workarounds (such as those listed here) that take very little time to put together and at least don’t fragment your composition.</p>
<p>One way of working around this is defining two constructors in your pages. The first, public parameterless (implicitly apparent when none are defined) and the second containing your dependencies. There is a <a href="https://blogs.msdn.microsoft.com/webdev/2016/10/19/modern-asp-net-web-forms-development-dependency-injection/">good article on MSDN</a> showing how you can then use an <a href="https://msdn.microsoft.com/en-us/library/system.web.ihttpmodule(v=vs.110).aspx"><code class="highlighter-rouge">IHttpModule</code></a> to hook your choice of container into the <a href="https://msdn.microsoft.com/en-us/library/system.web.httpapplication.prerequesthandlerexecute(v=vs.110).aspx"><code class="highlighter-rouge">PreRequestHandlerExecute</code></a> allowing you to take actions against the page being called prior to your <a href="https://msdn.microsoft.com/en-us/library/ms178472.aspx#lifecycle_events">page life cycle</a>.</p>
<p>I personally really like the new <a href="https://docs.microsoft.com/en-us/aspnet/core/fundamentals/dependency-injection"><code class="highlighter-rouge">Microsoft.Extensions.DependencyInjection</code></a> libraries so the following code shows an alternate version of the aforementioned MSDN article using the Microsoft library instead of <a href="https://autofac.org/">Autofac</a>.</p>
<p>Whether your <code class="highlighter-rouge">IHttpModule</code> acts as your <a href="http://blog.ploeh.dk/2011/07/28/CompositionRoot/">Composition Root</a> or whether it calls into it is up to you. For the sake of brevity, I’m going to add my service bindings in the module:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Collections.Concurrent</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Linq</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Reflection</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Web.UI</span><span class="p">;</span>
<span class="c1">// add your namespace</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">DependencyInjectionHttpModule</span> <span class="p">:</span> <span class="n">IHttpModule</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="n">IServiceProvider</span> <span class="n">Provider</span><span class="p">;</span>
<span class="k">private</span> <span class="k">static</span> <span class="k">readonly</span> <span class="n">ConcurrentDictionary</span><span class="p"><</span><span class="n">TypeInfo</span><span class="p">,</span> <span class="n">Action</span><span class="p"><</span><span class="n">Page</span><span class="p">>></span> <span class="n">PageConstructors</span>
<span class="p">=</span> <span class="k">new</span> <span class="n">ConcurrentDictionary</span><span class="p"><</span><span class="n">TypeInfo</span><span class="p">,</span> <span class="n">Action</span><span class="p"><</span><span class="n">Page</span><span class="p">>>();</span>
<span class="k">static</span> <span class="nf">DependencyInjectionHttpModule</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">services</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServiceCollection</span><span class="p">()</span>
<span class="p">.</span><span class="n">AddTransient</span><span class="p"><</span><span class="n">Foo</span><span class="p">,</span> <span class="n">Bar</span><span class="p">>();</span>
<span class="c1">// add more services or tie into composition root</span>
<span class="n">Provider</span> <span class="p">=</span> <span class="n">services</span><span class="p">.</span><span class="nf">BuildServiceProvider</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Init</span><span class="p">(</span><span class="n">HttpApplication</span> <span class="n">context</span><span class="p">)</span>
<span class="p">=></span> <span class="n">context</span><span class="p">.</span><span class="n">PreRequestHandlerExecute</span> <span class="p">+=</span> <span class="n">PreRequestHandlerExecute</span><span class="p">;</span>
<span class="k">private</span> <span class="k">void</span> <span class="nf">PreRequestHandlerExecute</span><span class="p">(</span><span class="kt">object</span> <span class="n">sender</span><span class="p">,</span> <span class="n">EventArgs</span> <span class="n">e</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">HttpContext</span><span class="p">.</span><span class="n">Current</span><span class="p">.</span><span class="n">CurrentHandler</span> <span class="k">is</span> <span class="n">Page</span> <span class="n">page</span><span class="p">)</span>
<span class="nf">ConstructorInject</span><span class="p">(</span><span class="n">page</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">void</span> <span class="nf">ConstructorInject</span><span class="p">(</span><span class="n">Page</span> <span class="n">page</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">pageTypeInfo</span> <span class="p">=</span> <span class="n">page</span><span class="p">.</span><span class="nf">GetType</span><span class="p">().</span><span class="n">BaseType</span><span class="p">.</span><span class="nf">GetTypeInfo</span><span class="p">();</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pageTypeInfo</span><span class="p">.</span><span class="n">IsAbstract</span><span class="p">)</span>
<span class="k">return</span><span class="p">;</span>
<span class="k">if</span> <span class="p">(</span><span class="n">PageConstructors</span><span class="p">.</span><span class="nf">ContainsKey</span><span class="p">(</span><span class="n">pageTypeInfo</span><span class="p">))</span>
<span class="n">PageConstructors</span><span class="p">[</span><span class="n">pageTypeInfo</span><span class="p">](</span><span class="n">page</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">constructors</span> <span class="p">=</span> <span class="n">pageTypeInfo</span><span class="p">.</span><span class="n">DeclaredConstructors</span>
<span class="p">.</span><span class="nf">Where</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="p">!</span><span class="n">c</span><span class="p">.</span><span class="n">IsStatic</span> <span class="p">&&</span> <span class="n">c</span><span class="p">.</span><span class="n">IsPublic</span><span class="p">)</span>
<span class="p">.</span><span class="nf">Select</span><span class="p">(</span><span class="n">c</span> <span class="p">=></span> <span class="k">new</span> <span class="nf">Constructor</span><span class="p">(</span><span class="n">c</span><span class="p">))</span>
<span class="p">.</span><span class="nf">ToArray</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">largestConstructor</span> <span class="p">=</span> <span class="n">constructors</span><span class="p">.</span><span class="nf">Max</span><span class="p">(</span><span class="n">x</span> <span class="p">=></span> <span class="n">x</span><span class="p">.</span><span class="n">Parameters</span><span class="p">.</span><span class="n">Length</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">constructors</span><span class="p">.</span><span class="nf">Count</span><span class="p">(</span><span class="n">x</span> <span class="p">=></span> <span class="n">x</span><span class="p">.</span><span class="n">Parameters</span><span class="p">.</span><span class="n">Length</span> <span class="p">==</span> <span class="n">largestConstructor</span><span class="p">)</span> <span class="p">!=</span> <span class="m">1</span><span class="p">)</span>
<span class="n">PageConstructors</span><span class="p">.</span><span class="nf">TryAdd</span><span class="p">(</span><span class="n">pageTypeInfo</span><span class="p">,</span> <span class="n">p</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"No suitable constructor could be found for </span><span class="p">{</span><span class="n">p</span><span class="p">.</span><span class="nf">GetType</span><span class="p">().</span><span class="n">AssemblyQualifiedName</span><span class="p">}</span><span class="s">."</span><span class="p">));</span>
<span class="k">else</span>
<span class="n">PageConstructors</span><span class="p">.</span><span class="nf">TryAdd</span><span class="p">(</span><span class="n">pageTypeInfo</span><span class="p">,</span> <span class="nf">Compose</span><span class="p">(</span><span class="n">page</span><span class="p">,</span> <span class="n">pageTypeInfo</span><span class="p">,</span> <span class="n">constructors</span><span class="p">.</span><span class="nf">Single</span><span class="p">(</span><span class="n">x</span> <span class="p">=></span> <span class="n">x</span><span class="p">.</span><span class="n">Parameters</span><span class="p">.</span><span class="n">Length</span> <span class="p">==</span> <span class="n">largestConstructor</span><span class="p">)));</span>
<span class="n">PageConstructors</span><span class="p">[</span><span class="n">pageTypeInfo</span><span class="p">](</span><span class="n">page</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">static</span> <span class="n">Action</span><span class="p"><</span><span class="n">Page</span><span class="p">></span> <span class="nf">Compose</span><span class="p">(</span><span class="n">Page</span> <span class="n">page</span><span class="p">,</span> <span class="n">TypeInfo</span> <span class="n">pageTypeInfo</span><span class="p">,</span> <span class="n">Constructor</span> <span class="n">constructor</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">parameters</span> <span class="p">=</span> <span class="k">new</span> <span class="kt">object</span><span class="p">[</span><span class="n">constructor</span><span class="p">.</span><span class="n">Parameters</span><span class="p">.</span><span class="n">Length</span><span class="p">];</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">var</span> <span class="n">i</span> <span class="p">=</span> <span class="m">0</span><span class="p">;</span> <span class="n">i</span> <span class="p"><</span> <span class="n">parameters</span><span class="p">.</span><span class="n">Length</span><span class="p">;</span> <span class="n">i</span><span class="p">++)</span>
<span class="n">parameters</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="p">=</span> <span class="n">Provider</span><span class="p">.</span><span class="nf">GetRequiredService</span><span class="p">(</span><span class="n">constructor</span><span class="p">.</span><span class="n">Parameters</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">ParameterType</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">InvalidOperationException</span> <span class="n">ex</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">p</span> <span class="p">=></span> <span class="k">throw</span> <span class="k">new</span> <span class="nf">InvalidOperationException</span><span class="p">(</span><span class="s">$"No suitable constructor could be found for </span><span class="p">{</span><span class="n">page</span><span class="p">.</span><span class="nf">GetType</span><span class="p">().</span><span class="n">AssemblyQualifiedName</span><span class="p">}</span><span class="s">."</span><span class="p">,</span> <span class="n">ex</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">p</span> <span class="p">=></span> <span class="n">constructor</span><span class="p">.</span><span class="n">MethodInfo</span><span class="p">.</span><span class="nf">Invoke</span><span class="p">(</span><span class="n">page</span><span class="p">,</span> <span class="n">parameters</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Dispose</span><span class="p">()</span> <span class="p">{</span> <span class="p">}</span>
<span class="k">private</span> <span class="k">class</span> <span class="nc">Constructor</span>
<span class="p">{</span>
<span class="k">public</span> <span class="n">ConstructorInfo</span> <span class="n">MethodInfo</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="n">ParameterInfo</span><span class="p">[]</span> <span class="n">Parameters</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">Constructor</span><span class="p">(</span><span class="n">ConstructorInfo</span> <span class="n">constructorInfo</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">MethodInfo</span> <span class="p">=</span> <span class="n">constructorInfo</span><span class="p">;</span>
<span class="n">Parameters</span> <span class="p">=</span> <span class="n">constructorInfo</span><span class="p">.</span><span class="nf">GetParameters</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This <code class="highlighter-rouge">IHttpModule</code> uses a <em>greedy</em> search to find the largest constructor on the Page you’re trying to instantiate, and then attempt to populate the parameters using the Microsoft Dependency Injection container. We can’t just ask the DI container to create the page itself because the Page will already have been created by the hidden away WebForms framework (the one that required there be a public parameterless constructor present), and in case you didn’t know, you don’t <em>have</em> to use the <code class="highlighter-rouge">new</code> keyword to call a constructor; it is just a method at the end of the day.</p>
<p>As a slight aside, many full DI container libraries implement an improved version of this <em>greedy</em> search in that they will attempt to find the largest constructor they have sufficient service bindings to actually fulfil. You’re welcome to expand the above snippet if you need that functionality (or ping me a comment below if you’re unsure how), but it should be fairly trivial to utilise the <code class="highlighter-rouge">IServiceCollection</code> to determine which constructors can and can not be fulfilled.</p>
<p>As with all <code class="highlighter-rouge">IHttpModules</code>, in order for this to be utilised at runtime you will need to add the correct entry to your <code class="highlighter-rouge">web.config</code>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
</pre></td><td class="rouge-code"><pre><span class="nt"><configuration></span>
<span class="nt"><system.web></span>
<span class="nt"><httpModules></span>
<span class="c"><!-- Use this if you are on IIS 7 and earlier --></span>
<span class="nt"><add</span> <span class="na">name=</span><span class="s">"DependencyInjectionHttpModule"</span> <span class="na">type=</span><span class="s">"Your.Namespace.DependencyInjectionHttpModule, Your.AssemblyName"</span><span class="nt">/></span>
<span class="nt"></httpModules></span>
<span class="nt"></system.web></span>
<span class="nt"><system.webServer></span>
<span class="nt"><modules></span>
<span class="c"><!-- Use this if you are on IIS 8 and later --></span>
<span class="nt"><add</span> <span class="na">name=</span><span class="s">"DependencyInjectionHttpModule"</span> <span class="na">type=</span><span class="s">"Your.Namespace.DependencyInjectionHttpModule, Your.AssemblyName"</span> <span class="nt">/></span>
<span class="nt"></modules></span>
<span class="nt"></system.webServer></span>
<span class="nt"></configuration></span>
</pre></td></tr></tbody></table></code></pre></div></div>
<blockquote>
<p>Make sure you replace <code class="highlighter-rouge">Your.Namespace</code> and <code class="highlighter-rouge">Your.AssemblyName</code>.</p>
</blockquote>
<h3 id="property-injection">Property Injection</h3>
<p>Property Injection is very similar to the constructor injection shown above. Instead of writing your own constructor to populate fields as shown above, your composition framework (be it imported or homebrew) becomes responsible for locating Properties on the page that need to be populated, and then populates them.</p>
<p>Generally, I really dislike this approach. By having dependencies passed to a constructor it’s very obvious that the class requires those dependencies in order to function (to the point that you can’t instantiate it without them). With properties however, it’s akin some very old (bad) practices that involved having to call <code class="highlighter-rouge">Initialise</code> or <code class="highlighter-rouge">Init</code> on a class before the class is available for use.</p>
<p>In my experience, it also leads to any dependency validation (null checking for example) being duplicated across the class, and there’s unless you add validation to the properties, implies that a user can modify your dependency properties (more null checking, concurrency issues, etc).</p>
<p>However, when all is said and done, property injection is the preferred approach when working with Web Forms purely because it looks a little cleaner. A good number of DI Containers such as <a href="https://cmatskas.com/dependency-injection-in-asp-net-webforms-with-structuremap/">StructureMap</a> and <a href="http://simpleinjector.readthedocs.io/en/latest/webformsintegration.html">SimpleInjector</a> can handle this property injection out of the box and be integrated into WebForms easily.</p>
<p>If you’d like to use the Microsoft DI Container shown in the previous section, you’re welcome to use this modified version of the above <code class="highlighter-rouge">IHttpModule</code> (sold as-is, no warranty…)</p>
<!--TODO: Add property injection alternative-->
<!--TODO: Add existing di module alternative (Unity)-->
<!--TODO: Add multi-target testing-->I recently started work for a new (and very exciting) company. Like many of my previous roles, this one entails a good mix of Brownfield and Greenfield development. I’ve met and worked with some great developers who would outright refuse roles that held too much Brownfield work, preferring pastures green, as well as working with some great devs that prefer the challenge of Brownfield.Kubernetes in Azure2017-12-04T00:00:00+00:002017-12-04T00:00:00+00:00http://blog.devbot.net/azure-k8s<p>There has been a lot of movement around kubernetes (k8s) of late. Several big name vendors have <a href="https://techcrunch.com/2017/09/20/kubernetes-gains-momentum-as-big-name-vendors-join-cloud-native-computing-foundation/">assumed k8s as the de facto standard</a>, and all the big name cloud providers now have some level of integration. I work at an Azure house (though we have a good chunk in AWS also) so we had been looking forward to getting involved with the <a href="https://azure.microsoft.com/en-gb/blog/introducing-azure-container-service-aks-managed-kubernetes-and-azure-container-registry-geo-replication/">new Azure Container Service</a>. However, the service (still in preview at time of writing) is far from ready. The myriad of bugs we encountered have lead us to deploying our own cluster in Azure to learn more about the procedure and technologies. We anticipate switching back to the managed service when we deem it to be ready, but for now I want to walk you along the journey we’ve taken in case you too would prefer the more hands on approach.</p>
<h2 id="azure-containers">Azure Containers</h2>
<p>For the time being, there are two versions of Azure Container Services. The original, publicly available and stable variant which adopted the acronym <em>ACS</em> as you would expect, though I believe is now referred to as <a href="https://azure.microsoft.com/en-gb/services/container-instances/">Container Instances</a>.</p>
<p>I took part in evaluating this service for my company at the time (several months ago), and decided the orchestration mechanics were lacking. Whilst it achieves the goal of running and scaling docker instances and load balancing them, there really wasn’t any other orchestration features available, and given the implementation seemed to be bespoke to Azure, extending the base featureset doesn’t appear to be viable.</p>
<p>The new Azure Container Service (AKS - the ‘K’ presumably denoting it’s Kubernetes integration) will, I’m sure, be a fantastic offering once it’s feature complete and stable. The premise of the service is that Azure will front the cost of the management nodes for you (utilising recently added k8s features that allows Azure to manage many clusters from their controllers).</p>
<p>However, the problems we encountered over a brief period:</p>
<ul>
<li>Whilst provisioning a <em>default</em> cluster is incredibly simple, customising it to any real degree is difficult, and often not possible at all (depending on level of customisation required).</li>
<li>Once provisioned, you don’t appear to be able to change much of the cluster configuration (node count, VM size, etc.).</li>
<li>The cheapest VM you can run nodes on is restricted to a <em>Standard D2 v2</em> when provisioning through AKS, so if you go with the default of 3 nodes, costs ~£220 p/month
<ul>
<li><em>This was a big deal for me; anything that falls outside ~£110 p/month means either massive teardown overhead or simply not being able to test and learn a tech using my MSDN Enterprise Azure Credits.</em></li>
</ul>
</li>
</ul>
<p>Fortunately, the AKS service is built upon the open-source <a href="https://github.com/Azure/acs-engine">ACS Engine</a>, which can be used standalone to produce the exact same mechanisms in a much more customisable manner. Personally, I also prefer this mechanism because the innards are not hidden away behind a featureless <em>Container Service</em> - instead you can see each VM, availability set, NSG, IP, NIC, etc.</p>
<h2 id="acs-engine">ACS Engine</h2>
<p>The ACS engine isn’t documented perfectly yet, but it’s certainly workable. My aim was to provision a multi-node cluster that met my cost requirements (£110 p/month) whilst providing all the features possible for me to play with.</p>
<p>The following is a walkthrough on how to get setup. The steps can be completed on Linux or Windows, through Powershell or Bash. To where you install the pre-requisites will determine where you run the remaining steps.</p>
<h3 id="prerequisites">Prerequisites</h3>
<ul>
<li>Install <a href="https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest">Azure CLI 2.0</a></li>
<li>Install <a href="https://kubernetes.io/docs/tasks/tools/install-kubectl/">Kubectl</a></li>
</ul>
<h3 id="getting-started">Getting Started</h3>
<p>For the most part, I followed the <em>ACS Engine</em> advice on how to install the <a href="https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/deploy.md#acs-engine-the-long-way">long way</a>. Following the simple <code class="highlighter-rouge">acs engine deploy</code> mechanism will create all the components the normal AKS service would have done with little customisation (though at least not hidden behind the abstract service).</p>
<p>In order to attain greater flexibility, you’ll want to follow this slightly lengthier process:</p>
<h4 id="1-login">1. Login</h4>
<p>Open your choice of shell and get yourself logged into the Azure CLI.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az login
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This will typically result in a short trip to the specified URL where you’ll need to enter the code displayed in the shell.</p>
<h4 id="2-resource-group">2. Resource Group</h4>
<p>You’ll want to create a resource group to contain all the moving pieces (there are quite a large number). Here I’m creating a resource group called <code class="highlighter-rouge">furture-digital-aks</code> in West Europe.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az group create <span class="nt">-l</span> westeurope <span class="nt">-n</span> future-digital-aks
</pre></td></tr></tbody></table></code></pre></div></div>
<h4 id="3-key-vault">3. Key Vault</h4>
<p>Key Vault will be used to store the credentials for your service principal, which in turn is responsible for managing your azure resources in accordance with Kubernetes requirements. You can skip key vault and use plain old credentials if you prefer.</p>
<p>This command will create a key vault called <code class="highlighter-rouge">future-digital-aks-kv</code> in the resource group we just created, again, located in West Europe (I recommend keeping related resources co-located!). I’ve also enabled all the features - as far as I know, it doesn’t cost to have them enabled and you never know when they might come in handy.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az keyvault create <span class="nt">-n</span> future-digital-aks-kv <span class="nt">-g</span> future-digital-aks <span class="nt">--enable-soft-delete</span> <span class="nb">true</span> <span class="nt">--enabled-for-deployment</span> <span class="nb">true</span> <span class="nt">--enabled-for-disk-encryption</span> <span class="nb">true</span> <span class="nt">--enabled-for-template-deployment</span> <span class="nb">true</span> <span class="nt">-l</span> westeurope
</pre></td></tr></tbody></table></code></pre></div></div>
<h4 id="4-certificate">4. Certificate</h4>
<p>With key vault available, I now need to create a certificate for our service principal to use. Nothing fancy here, simple create a certificate called <code class="highlighter-rouge">aks-sp-cert</code> using the default cert policy.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az keyvault certificate create <span class="nt">--vault-name</span> future-digital-kv <span class="nt">-n</span> aks-sp-cert <span class="nt">-p</span> <span class="s2">"</span><span class="si">$(</span>az keyvault certificate get-default-policy<span class="si">)</span><span class="s2">"</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<h4 id="5-service-principal">5. Service Principal</h4>
<p>With the certificate ready, we can create our service principal. In this example, I’ve scoped the service principal so that it only has it’s default role of <code class="highlighter-rouge">Contributor</code> within our newly created resource group and named it <code class="highlighter-rouge">aks</code>. Notice we provide the name of the certificate created in the previous step, <code class="highlighter-rouge">aks-sp-cert</code>, and provide details for the key vault created.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az ad sp create-for-rbac <span class="nt">-n</span> aks <span class="nt">--scopes</span> /subscriptions/<YOUR SUBSCRIPTION ID>/resourceGroups/future-digital-aks <span class="nt">--cert</span> aks-sp-cert <span class="nt">--keyvault</span> future-digital-kv
</pre></td></tr></tbody></table></code></pre></div></div>
<p><strong>Important</strong>: take note of the <code class="highlighter-rouge">appId</code> in the response to this command, you will need it later.</p>
<p><img src="../images/k8s-sp-appid.png" alt="service principal id" /></p>
<blockquote>
<p><strong>Note</strong>: <em>You’ll need to input your own subscription ID if you want to scope this service principal. Alternatively, you can remove the <code class="highlighter-rouge">--scope</code> switch if you prefer.</em></p>
</blockquote>
<h4 id="6-ssh-key-pair">6. SSH Key Pair</h4>
<p>The virtual machines created for the cluster will be running on Linux, which means any administration required on the boxes directly will require <a href="https://en.wikipedia.org/wiki/Secure_Shell">SSH</a>. <em>ACS Engine</em> will setup your machines for SSH, but in order for you to connect you will need to create an SSH Keypair. There are several ways to accomplish this, but from bash (available in Powershell if you’re using <a href="https://msdn.microsoft.com/en-gb/commandline/wsl/about">WSL in Windows 10</a>), I used the following:</p>
<h5 id="61-create-private-key">6.1. Create Private Key</h5>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>ssh-keygen <span class="nt">-C</span> <span class="s2">"your@email.com"</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>By default, this will create a 4096-bit RSA key, which should be perfectly sufficient. You will be prompted as to where the generated key should be stored (<code class="highlighter-rouge">~/.ssh/id_rsa</code> by default) and what keyphrase/password to use.</p>
<h5 id="62-export-public-key">6.2 Export Public Key</h5>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>ssh-keygen <span class="nt">-e</span> <span class="nt">-f</span> ~/.ssh/id_rsa.pub
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This will create an <code class="highlighter-rouge">id_rsa.pub</code> file containing your public key in the default <code class="highlighter-rouge">RFC4716</code> format (you can use <code class="highlighter-rouge">-m</code> to specify alternative formats ike <code class="highlighter-rouge">PKCS8</code> and <code class="highlighter-rouge">PEM</code> should other tooling require such).</p>
<p>You can simply use <code class="highlighter-rouge">cat ~/.ssh/id_rsa.pub</code> to display the contents of the public key file which can be copied elsewhere as required.</p>
<blockquote>
<p><strong>Note</strong>: <em>You’ll need this exported public key in the next step.</em></p>
</blockquote>
<h4 id="7-cluster-definition">7. Cluster Definition</h4>
<p>We have now prepared all the resources we require to get a cluster up and running. The next step is to create/modify a cluster definition for the <em>ACS Engine</em>. You can find the <a href="https://github.com/Azure/acs-engine/blob/master/examples/kubernetes.json">default example here</a> or you’re welcome to work from the one I’m using in this example:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
</pre></td><td class="rouge-code"><pre><span class="p">{</span><span class="w">
</span><span class="nl">"apiVersion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"vlabs"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"orchestratorProfile"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"orchestratorType"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Kubernetes"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"masterProfile"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"count"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
</span><span class="nl">"dnsPrefix"</span><span class="p">:</span><span class="w"> </span><span class="s2">"future-digital-k8-weu"</span><span class="p">,</span><span class="w">
</span><span class="nl">"vmSize"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Standard_B2s"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"agentPoolProfiles"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"agentpool1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"count"</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">,</span><span class="w">
</span><span class="nl">"vmSize"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Standard_B1s"</span><span class="p">,</span><span class="w">
</span><span class="nl">"availabilityProfile"</span><span class="p">:</span><span class="w"> </span><span class="s2">"AvailabilitySet"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="nl">"linuxProfile"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"adminUsername"</span><span class="p">:</span><span class="w"> </span><span class="s2">"azureuser"</span><span class="p">,</span><span class="w">
</span><span class="nl">"ssh"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"publicKeys"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"keyData"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ssh-rsa AAAAB3N...<snipped>...@futuredigital.co.uk"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"servicePrincipalProfile"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"clientId"</span><span class="p">:</span><span class="w"> </span><span class="s2">"3b7ab5c1-b450-49a7-9c43-e0e0b4adf887"</span><span class="p">,</span><span class="w">
</span><span class="nl">"secret"</span><span class="p">:</span><span class="w"> </span><span class="s2">"aks-sp-cert"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></pre></td></tr></tbody></table></code></pre></div></div>
<blockquote>
<p><strong>Note</strong>: <em>You must choose your own custom <code class="highlighter-rouge">dnsPrefix</code> within the <code class="highlighter-rouge">masterProfile</code> segment.</em></p>
</blockquote>
<p>You’ll need to inject several pieces of information into the json that you’ll have gathered over the previous steps:</p>
<ul>
<li><code class="highlighter-rouge">properties.linuxProfile.ssh.publicKeys.keyData</code> - this is where the public key from <a href="#62-export-public-key">6.2 Export Public Key</a> needs to go.</li>
<li><code class="highlighter-rouge">properties.servicePrincipalProfile.clientId</code> - this is where you put the <code class="highlighter-rouge">appId</code> from <a href="#5-service-principal">5. Service Principal</a>.</li>
<li><code class="highlighter-rouge">properties.servicePrincipalProfile.secret</code> - simply insert the <em>name</em> of the certificate created for the service principal.</li>
</ul>
<p>You can from here adjust master and node VM sizes and counts, and set your own linux user name if you prefer.</p>
<h4 id="8-generate-templates">8. Generate Templates</h4>
<p>We will now instruct the <em>ACS Engine</em> to generate an Azure Resource Management template for us to deploy. This step will simply create the json required for our deployment, which you can futher edit if you so wish:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>acs-engine generate path/to/kubernetes.json
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The command will create an <code class="highlighter-rouge">_output</code> directory which will contain a directory matching your <code class="highlighter-rouge">dnsPrefix</code>. Inside here are all the ARM template files.</p>
<h4 id="9-deploy-template">9. Deploy Template</h4>
<p>Once you’re satisfied with the ARM template, simply deploy to azure:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre>az group deployment create <span class="nt">-n</span> aks-deployment <span class="nt">-g</span> future-digital-aks <span class="nt">--template-file</span> .<span class="se">\_</span>output<span class="se">\y</span>our-dns-prefix<span class="se">\a</span>zuredeploy.json <span class="nt">--parameters</span> .<span class="se">\_</span>output<span class="se">\y</span>our-dns-prefix<span class="se">\a</span>zuredeploy.parameters.json
</pre></td></tr></tbody></table></code></pre></div></div>
<p>This will take quite a while to execute, but once complete, you should have an AKS cluster something like the following</p>
<p><img src="../images/k8s-cluster-resources.png" alt="aks deployed resources" /></p>
<h2 id="next-steps">Next Steps</h2>
<p>I recommend checking out this <a href="https://github.com/Azure/acs-engine/blob/master/docs/kubernetes/walkthrough.md">Kubernetes Walkthrough</a>.</p>There has been a lot of movement around kubernetes (k8s) of late. Several big name vendors have assumed k8s as the de facto standard, and all the big name cloud providers now have some level of integration. I work at an Azure house (though we have a good chunk in AWS also) so we had been looking forward to getting involved with the new Azure Container Service. However, the service (still in preview at time of writing) is far from ready. The myriad of bugs we encountered have lead us to deploying our own cluster in Azure to learn more about the procedure and technologies. We anticipate switching back to the managed service when we deem it to be ready, but for now I want to walk you along the journey we’ve taken in case you too would prefer the more hands on approach.Clean Space - Neural Networks2017-10-11T00:00:00+00:002017-10-11T00:00:00+00:00http://blog.devbot.net/neural-networks<p>It’s been more than a year since I last talked about <a href="http://blog.devbot.net/clean-space-game/">Clean Space</a>, so I wanted to throw out this (lengthy!) post with a bit of a status update, as well as talk about something I’m working on with regards to <em>Clean Space</em> that others might find interesting. So there’s the first bit of information - work on <em>Clean Space</em> continues. Not as actively as I would like, but I have to balance <em>work</em> and <em>living</em>.</p>
<p style="font-size: 12px; text-align: center;"><img src="../images/clean-space-drone.jpg" alt="clean space drone" />
<em>Sensor Projection of Unidentified Vessel <code class="highlighter-rouge">MTRD-4482</code></em></p>
<p>I won’t say the game has come on leaps and bounds, it really hasn’t. I spent a long time getting annoyed at game frameworks for having truly awful APIs, even forked MonoGame twice to give it a decent Game Loop, DI and an eventing backbone before giving up on the framework completely (<em>I actually have an incomplete blog post about this which I can go back and complete if someone expresses some interest</em>). My current plan is to use <a href="https://threejs.org/">ThreeJS</a> for rendering which looks plenty capable, and supports the web application model required by the massively intensive processing requirements of certain game aspects.</p>
<!-- markdownlint-disable MD007 -->
<!-- markdownlint-disable MD010 -->
<!-- TOC -->
<ul>
<li><a href="#artificial-intelligence">Artificial Intelligence</a></li>
<li><a href="#goals">Goals</a>
<ul>
<li><a href="#layers-in-your-layers">Layers in your Layers</a>
<ul>
<li><a href="#engineering">Engineering</a></li>
<li><a href="#astrogation">Astrogation</a></li>
<li><a href="#captain">Captain</a></li>
</ul>
</li>
<li><a href="#breakdown">Breakdown</a></li>
</ul>
</li>
<li><a href="#target">Target</a>
<ul>
<li><a href="#basics">Basics</a></li>
</ul>
</li>
</ul>
<!-- /TOC -->
<!-- markdownlint-enable MD007 -->
<!-- markdownlint-enable MD010 -->
<h2 id="artificial-intelligence">Artificial Intelligence</h2>
<p>If you have a better memory than I, you might recall 12-18 months ago that I explained how comprehensive the AI in <em>Clean Space</em> would be. The goal was “simple”, <em>every</em> single thing a Human would be able to do in-game, an AI would also be able to do, and vice versa. The AI(s) would have no more game state made available to it than a player, and <a href="http://blog.devbot.net/clean-space-game/#perspective"><em>perspective</em></a> would be forced upon player and AI alike.</p>
<p>A lot of AI-like behaviour can, and often is implemented in the form of numerous rules that react to events. However, because I’m apparently glutton for punishment, and as the title of this article probably gave away, I’m not taking the easy way out. Instead, I’m using as close to real artificial intelligence as I can get, which means neural networks and deep learning.</p>
<h2 id="goals">Goals</h2>
<p>The high level goals of Clean Space have never changed.</p>
<ul>
<li>I want a game that <em>I</em> want to play. If anyone else happens to enjoy it too, great. If I can make enough money from it to pay server costs, awesome. If I can make enough money from it to buy a GTR… one can dream.</li>
<li>I want to learn a metric sh*t tonne. There are so many aspects to this game that it’s literally impossible for any single person to know them all; I want exposure to as much of that as I can.</li>
<li>I want to set an example. I wrote an article some time ago about how I feel about the <a href="http://blog.devbot.net/industry/">Game Industry</a>. The software industry as a whole isn’t much better, but what’s <em>exposed</em> of the Gaming side is worse comparably, in my opinion.</li>
</ul>
<p>Clearly, meeting any kind of timescales is not an example I’ll be setting. But I hope to provide some clean practices and perhaps frameworks, framework/engine integration, tech articles on <em>Clean Space</em> mechanics, and of course <em>Clean Engine</em>.</p>
<p>I haven’t defined a specific <em>minimum viable product</em> for <em>Clean Space</em>, because despite the talk about money above, I don’t care all too much if the game ever releases. It’ll come along, or not, when it’s good and ready. But that does mean progressing towards smaller goals and trying to keep up with an ever-evolving game design.</p>
<p>In this article, I want to focus on AI and show how it contributes towards the bigger picture.</p>
<h3 id="layers-in-your-layers">Layers in your Layers</h3>
<p><img src="../images/ai-in-ai-meme.jpg" alt="AI in your AI meme" /></p>
<p>There are so many AI’s required to fulfill my vision for <em>Clean Space</em>, that I’m going to have to name them for this to next section to make sense. I’ll apologise in advance for constant interruptions to immersion as I explain how the AI vs. Player mechanics will work.</p>
<p>Let’s start our example with a space craft, in a stellar system. I’m going to jump straight to the vehicle’s AI and skip over all the AI involved in construction, crew allocation, objective creation and assignment, etc.</p>
<h4 id="engineering">Engineering</h4>
<p>Let’s say the space craft has 2 primary propulsion units (main drives, or engines if you prefer). The Chief Engineer, in addition to maintaining everything, has been plotting the characteristics of each engine. She knows that every engine has it’s own quirks, derived from the materials used, assembly quality, fuel purity, and so forth. These hidden variables no one but the game engine knows completely (which means the AI doesn’t know them either). However, by monitoring data that <em>is</em> available (temperatures, pressures, harmonics, specific impulse, acceleration curves, etc.) a skilled engineer will be able to predict to some degree of accuracy how the engines will behave when they’re put to various uses in the future.</p>
<p>Personally, I’m not the kind of guy that would enjoy plotting this data in a spreadsheet and speculating as to the maths. I really hope one day that a niche within a <em>Clean Space Community</em> does attempt this though. I’d be fascinated as to how close they could get to the actual implementation, which I’ll try to make complex and elusive enough so as to make it impossible to predict with 100% certainty. Players won’t be <em>punished</em> for not undertaking these tasks themselves though, the AI will truly be the best I can make it. It won’t have access to anything the player can’t get hold of, so it will be working within a perspective without visibility of the myriad of hidden variables that actually drive the calculations, but predictive machine learning should be very useful here.</p>
<h4 id="astrogation">Astrogation</h4>
<p>Constant revisions by the engineer are being fed into astrogation. There are many ways to get from A to B in space. In modern day technology we often see amazing trajectories like this one (<a href="https://www.asteroidmission.org/">OSIRIS-REx</a>):</p>
<p><img src="https://www.gifgif.io/qZrtE9.gif" alt="osiris rex flight path" /></p>
<p>Ignoring that this trajectory isn’t intended to get from A-B per se, it does make for a good example of how modern day space flight paths rely on very limited specific impulse and delta-v. Given <em>Clean Space</em> works with more futuristic technologies and much lowered constraints, getting from A-B doesn’t need to follow these spectactular orbital trajectories.</p>
<p>As such, there are several flight profiles available to an astrogator, which in turn were derived from machine learning, not necesarrily pure unadulterated maths. The astrogator is responsible for feeding space craft characteristics such as mass, and the engineer’s recommendations for drive profiles, and then teaching her machine learning algorithms how to plot a course. Profiles might be trained by rewarding the conservation of fuel or time, reduction of strain on equipment, accuracy of target orbit, accuracy meeting a target rendevous epoch, or whichever reward weights the astrogator chooses. Ultimately, the astrogator is trying to prepare her virtual toolbelt so that no matter where the Captain wants to go and how they want to get there, the astrogator is ready to make that happen.</p>
<p>Processing time won’t be unlimited, so prioritising which profiles are trained best and kept up to date will require the astrogator to understand the Captain’s objectives and whims.</p>
<p>In-flight, the astrogator will also need to account for impromptu equipment failures and disparity between predicted and actual drive profiles, not to mention she may not have all necessary orbital data to have correctly accounted for relativistic effects, so will need to be able to compensate and adapt.</p>
<h4 id="captain">Captain</h4>
<p><em>Clean Space</em> is about as <em>sandbox, open-world</em> as you can get. But, whilst you <em>can</em> do anything you want, that doesn’t mean you’ll be <em>rewarded</em> for doing so. Planets, governments, syndicates, corporations, etc. are trying to balance a huge number of responsiblities such as economics and security. There is no magic cash cow in <em>Clean Space</em> that swings a wand and introduces money into the economy everytime you shoot a magically spawned pirate space craft. That money has to come from <em>somewhere</em> (as would the pirate…).</p>
<p>Captain’s will have spent ample time learning other trades and have a good understanding of roles available to space craft (no insurer would allow an inexperienced unqualified plebeian to take control of such an expensive piece of equipment, nor would any player have the startup cash to outright buy such a vehicle). Which role the captain and vehicle fill will be a matter of preference and supply & demand. Not <em>everyone</em> can be a Military Captain if there is insufficient drive for security - the government will reduce the budget and the admiralty will reduce expenditure (including rewards), effectively prohibiting the expansion (or forcing a reduction in) Military capable vehicles.</p>
<p>The game engine will attempt to balance player short falls by introducing AI to fulfill roles players don’t want, but if there is no demand for a role, a player is going to have an all but impossible time filling said role.</p>
<p>As such, the Captain in our above scenario has objectives, some kind of role they understand and know how to be rewarded. The Captain has to rely upon all the other layers of AI above in order to fulfill those objectives, for example, as Search & Rescue, plotting a least time course to a vehicle in distress in order to collect a reward from the distressed space craft’s insurer or be compensated by the government for that role. Or, as a gargantuan hauler moving materials between ore processing plants and refined material factories.</p>
<p>Whatever the purpose, whether the Captain is played by an actual human or an AI, it’s unavoidable that somewhere along the chain, they’ll be reliant upon AI. Layers of them!</p>
<h3 id="breakdown">Breakdown</h3>
<p>In order to achieve the above, the goal needs to be broken down into much smaller, simpler stages. There is no single AI, that I know of, that could do all of the above, and if there was, I doubt I could afford the ongoing server costs of keeping such a monster fed.</p>
<p>But, if we examine the smaller objectives of the AI layers, much more achievable goals can be set. The predictive machine learning required of the engineer predicting drive characteristics is quite a well known problem nowadays, so I’m going to gloss over that one.</p>
<p>The astrogation profiles however, that’s something I haven’t accomplished yet. I don’t even know if it’s possible. The rest of this article will detail my latest foray into neural networks tackling orbital mechanics.</p>
<p>Wish me luck…</p>
<h2 id="target">Target</h2>
<p>At a very high level, what I’m aiming for is the following:</p>
<p><img src="../images/clean-space-neural-network-astrogation-01.jpg" alt="astrogation neural network" /></p>
<p>Information relating to the current orbit of the vehicle and target orbit can be represented by <a href="https://en.wikipedia.org/wiki/Orbital_elements">orbital elements</a> along with information regarding the equipment. A great many other factors may also need to be accounted for such as other orbital bodies, but we’ll aim for this, for now.</p>
<p>The intent is to feed all this information into the blackbox that is the neural network, which can spit out drive usage (i.e. when to use the drives, how much to use them, for how long, and in which direction). As was demonstrated in my <a href="http://blog.devbot.net/clean-space-lunar-orbit">lunar orbit article</a>, it is often necessary to perform multiple <em>burns</em> in order to achieve a desired orbit, so it would be necessary for our drive usage to be able to be fed into a function that can predict the new orbit, which in turn could be passed back to the start to allow the network(s) to further refine the orbit.</p>
<p>Now, for those of you that know anything about machine learning and neural networks, it’s probably obvious that there are several flaws in either this high level plan, and even then, it is by no means a simple thing to achieve, even for an expert.</p>
<p>I am no expert.</p>
<h3 id="basics">Basics</h3>
<p>Thanks to authors like <a href="https://stevenmiller888.github.io">Steven Miller</a> and <a href="http://www.robosoup.com/category/neural-network">John Wakefield</a>, plebs with no background or higher education (like me) have access to some great resources.</p>
<p>I know from reading through these and many other articles on the subject, that I should probably pick an existing framework. However, I won’t. Not because of any <em>Not-Invented-Here</em> stance, but because I really need to get to grips with how these mechanisms work. I don’t <em>want</em> to write a framework, but I <em>do</em> want to learn how they work.</p>
<p>So, my first steps are to create a framework that can handle the most simple of neural networks, the <em>Single Layer Perceptron</em>. To prove the fundamentals of my framework, I want to start with an easy challenge like that proposed by <a href="https://github.com/trentsartain">Trent Sartain</a>, to create a network that can <a href="https://github.com/trentsartain/Neural-Network#what-are-the-parts-of-a-neural-network">solve <em>Exclusive Or (XOR)</em></a>.</p>
<p>I will do this by following <em>John Wakefields</em> tutorial series in order to progress solving <em>Trent Sartain’s</em> <code class="highlighter-rouge">XOR</code> challenge until the network requires something more difficult to work out. I won’t be copy/pasting any code, instead I’ll try to understand each phase of progress and then re-implement that myself (the reason for doing so is that, quite often, these very clever data scientists know enough about programming to make something functional, but their code isn’t as <em>clean</em> or even <em>OOP</em> as I would normally code, so I prefer to <em>translate</em> for my own sanity).</p>
<p>Speaking of OOP, having read a dozen articles I’ve already planned ahead somewhat, so if the following seems a bit overkill or the design decisions dubious, bear with it. I think they’ll be the right choice looking ahead slightly, as non-agile as that might be. I’ll start with the following structure:</p>
<p><img src="../images/neural-network-solution-01.png" alt="neural network solution" /></p>
<p>My intent is to have the key services exposed within the <code class="highlighter-rouge">CleanSpace.NeuralNetwork</code> project and associated tests in the <code class="highlighter-rouge">CleanSpace.NeuralNetwork.Tests</code> project. The idea behind the <code class="highlighter-rouge">CleanSpace.NeuralNetwork.AspNetCore</code> is to provide integration with ASP.NET Core, providing middleware that can handle varying types of operations.</p>
<p>Having read a great number of articles and sample applications dealing with neural networks, one of my biggest code gripes is the configuration models the samples and frameworks exposed. Due to the complexity of neural networks, I’m going to avoid passing in magic numbers and strings as well as multi-dimension arrays and any manner of typically avoided code, in favour of a Fluent API. If I do have to provide strange arrays and such, the code will hopefully make it much clearer what they are.</p>
<p>My first test will be an attempt to represent and prove John’s simplest network:</p>
<p><img src="../images/neural-network-robosoup-01.png" alt="robosoup simplest network" /></p>
<p>As you can see, there is no mysterious <em>hidden layer</em> in this network. It’s straight forward input to output in order to prove the code. This first test will try to train the network to always output <code class="highlighter-rouge">0</code>, regardless of the input (as per <a href="https://stevenmiller888.github.io/mind-how-to-build-a-neural-network/">Steven’s example</a>).</p>
<p>The first problem is that networks rely upon a random distribution during initialisation. For those of you with any experience with automated testings, having random numbers flow through the test is a nightmare because it may cause the test to intermittently fail, reducing confidence in the test. To get around this, I’ll use a seeded random with a known seed (such as zero). That will give me a predictable / reproducable random distribution.</p>
<p>To represent the above network with a fluent API, I like to start with the Fluent Description first. I start with a fixture:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="k">namespace</span> <span class="nn">CleanSpace.NeuralNetwork.Tests</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">ApplicationFixture</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">IServiceProvider</span> <span class="n">_provider</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">ApplicationFixture</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">_provider</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServiceCollection</span><span class="p">()</span>
<span class="p">.</span><span class="nf">AddNeuralNetwork</span><span class="p">()</span>
<span class="p">.</span><span class="nf">BuildServiceProvider</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">NeuralNetworkBuilder</span> <span class="nf">CreateNetwork</span><span class="p">()</span>
<span class="p">=></span> <span class="n">_provider</span><span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">NeuralNetworkBuilder</span><span class="p">>();</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The fixture being injected is very simple; I’m just using it as a composition root and abstract factory for the <code class="highlighter-rouge">NeuralNetworkBuilder</code> (via <em>service location</em> seeing as test projects don’t have a dependency injection bootstrapper like ASP.NET Core does). The test class itself is a bit more complicated, but its a test pattern I use for more complex projects:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">FluentAssertions</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System.Threading.Tasks</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Xunit</span><span class="p">;</span>
<span class="k">namespace</span> <span class="nn">CleanSpace.NeuralNetwork.Tests</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">SingleLayerPerceptronTests</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Given2RandomInputs</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Random</span> <span class="n">_random</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Random</span><span class="p">(</span><span class="m">0</span><span class="p">);</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">NeuralNetworkBuilder</span> <span class="n">_network</span><span class="p">;</span>
<span class="k">private</span> <span class="nf">Given2RandomInputs</span><span class="p">(</span><span class="n">ApplicationFixture</span> <span class="n">fixture</span><span class="p">)</span> <span class="p">=></span> <span class="n">_network</span> <span class="p">=</span> <span class="n">fixture</span>
<span class="p">.</span><span class="nf">CreateNetwork</span><span class="p">()</span>
<span class="p">.</span><span class="nf">AddInputs</span><span class="p">(</span><span class="n">_random</span><span class="p">.</span><span class="nf">NextDouble</span><span class="p">(),</span> <span class="n">_random</span><span class="p">.</span><span class="nf">NextDouble</span><span class="p">())</span>
<span class="p">.</span><span class="nf">WithForwardPropagation</span><span class="p">();</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">WhenTargetIsZero</span> <span class="p">:</span> <span class="n">Given2RandomInputs</span><span class="p">,</span> <span class="n">IClassFixture</span><span class="p"><</span><span class="n">ApplicationFixture</span><span class="p">></span>
<span class="p">{</span>
<span class="k">public</span> <span class="nf">WhenTargetIsZero</span><span class="p">(</span><span class="n">ApplicationFixture</span> <span class="n">fixture</span><span class="p">)</span> <span class="p">:</span> <span class="k">base</span><span class="p">(</span><span class="n">fixture</span><span class="p">)</span> <span class="p">=></span> <span class="n">_network</span>
<span class="p">.</span><span class="n">AddOutput</span><span class="p"><</span><span class="kt">double</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">WithBackPropagation</span><span class="p">()</span>
<span class="p">.</span><span class="nf">Targeting</span><span class="p">(</span><span class="m">0</span><span class="p">);</span>
<span class="p">[</span><span class="n">Fact</span><span class="p">]</span>
<span class="k">public</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">ThenEachIterationReducesMarginOfError</span><span class="p">()</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">firstPass</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_network</span><span class="p">.</span><span class="nf">ExecuteOnce</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">secondPass</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_network</span><span class="p">.</span><span class="nf">ExecuteOnce</span><span class="p">();</span>
<span class="n">secondPass</span><span class="p">.</span><span class="n">MarginOfError</span>
<span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">BeLessThan</span><span class="p">(</span><span class="n">firstPass</span><span class="p">.</span><span class="n">MarginOfError</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>As you can see, I use <a href="https://xunit.github.io/docs/shared-context.html">xUnit Fixtures</a> to pass in my <code class="highlighter-rouge">ApplicationFixture</code> (composition root) and hide some of that unimportant noise out of the way. I then build up a traditional <em>Given-When-Then</em> structure via classes. The important pieces add up to the following:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
</pre></td><td class="rouge-code"><pre><span class="n">_network</span> <span class="p">=</span> <span class="n">_neuralNetworkBuilder</span>
<span class="p">.</span><span class="nf">CreateNetwork</span><span class="p">()</span>
<span class="p">.</span><span class="nf">AddInputs</span><span class="p">(</span><span class="n">_random</span><span class="p">.</span><span class="nf">NextDouble</span><span class="p">(),</span> <span class="n">_random</span><span class="p">.</span><span class="nf">NextDouble</span><span class="p">())</span>
<span class="p">.</span><span class="nf">WithForwardPropagation</span><span class="p">()</span>
<span class="p">.</span><span class="n">AddOutput</span><span class="p"><</span><span class="kt">double</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">WithBackPropagation</span><span class="p">()</span>
<span class="p">.</span><span class="nf">Targeting</span><span class="p">(</span><span class="m">0</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">firstPass</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_network</span><span class="p">.</span><span class="nf">ExecuteOnce</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">secondPass</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_network</span><span class="p">.</span><span class="nf">ExecuteOnce</span><span class="p">();</span>
<span class="n">secondPass</span><span class="p">.</span><span class="n">MarginOfError</span>
<span class="p">.</span><span class="nf">Should</span><span class="p">().</span><span class="nf">BeLessThan</span><span class="p">(</span><span class="n">firstPass</span><span class="p">.</span><span class="n">MarginOfError</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>As you can see, I’m using a Fluent API to build up the network, then executing the network a couple of times and asserting that the <code class="highlighter-rouge">MarginOfError</code> is reducing after each iteration. As a <a href="https://en.wikipedia.org/wiki/Test-driven_development">TDD</a> developer, I wrote all this test code without having created any implementation, so I had nothing but compilation errors at that stage.</p>
<p>Creating the Fluent API implementation is a bit tedious but given I’ve done them before, I avoided some of the common pitfalls. The observant among you may have noticed that I haven’t given any details in the test for <em>how</em> forward and back propagation should work. That’s because I want to provide some basic defaults based on the aforementioned neural network tutorials, and later expand the core library with additional implementations and provide extensibility for custom implementations.</p>
<p>All of the above can be passed by creating some empty methods to satisfy the compilation errors and then setting <code class="highlighter-rouge">MarginOfError</code> to the next number in a decrementing sequence each time <code class="highlighter-rouge">ExecuteOnce</code> is called. I had to add a load of unit tests to flesh out some more thorough behaviour which I won’t detail in piecemeal here (you can go <a href="https://github.com/smudge202/neural-network">check out the code</a> if you want nitty gritty detail).</p>
<p>Clearly, the network doesn’t have a hidden layer in the above network, so I wanted to make the code extensible enough to have one or more hidden layers without going beyond the realm of the test. Fortunately, following <a href="https://en.wikipedia.org/wiki/SOLID_(object-oriented_design)">SOLID</a> kept things on track and saved me some lengthy refactoring later.</p>It’s been more than a year since I last talked about Clean Space, so I wanted to throw out this (lengthy!) post with a bit of a status update, as well as talk about something I’m working on with regards to Clean Space that others might find interesting. So there’s the first bit of information - work on Clean Space continues. Not as actively as I would like, but I have to balance work and living.Stumped by Simple Technical Test2017-09-30T00:00:00+00:002017-09-30T00:00:00+00:00http://blog.devbot.net/stumped-by-simple<p>I recently went back out to market in search of a new role and was quickly put on to a really exciting company. The outcome to my application to this company is not yet determined, but part of the process is to complete what appears at first glance to be a simple technical test. However, having given the test a bit of thought throughout the course of the day, it’s actually a <em>lot</em> more complicated than it first appears. In fact, I spent quite a while completely stumped as to a satisfactory design and implementation. I’m not sure at this stage whether this was intentional, some sneaky derivative of some known difficult problem that, having never completed college or university I’m unaware of, or if I was just over-thinking matters.</p>
<p>Anyway. I’m going to put to paper, so to speak, my thought process as I walk through this test. I’m hoping it will help me derive, refine and justify a solution, whilst also proving of interest to you. Unfortunately, I won’t be able to publish this article without permission from the company that issued the test, but hopefully this article makes the light of day at some point in the future.</p>
<p><img src="../images/funny-stuck.jpg" alt="funny stuck cow" /></p>
<h2 id="requirements">Requirements</h2>
<p>In essence, from a pool of 10 resources, the intent is to select 2 resources at random to complete half day shifts, per day, whilst complying with the following rules:</p>
<ol>
<li>A resource can do at most one half day shift in a day.</li>
<li>A resource cannot have half day shifts on consecutive days.</li>
<li>Each resource should have completed two half day shifts in any 2 week period.</li>
</ol>
<p>Those were the business requirements (re-worded slightly), as I received them. Immediately I note that the second requirement supercedes and makes superfluous the first requirement. By not allowing the same resource to be allocated to more than one shift in any 2 day period, a resource won’t be able to have 2 shifts in a single day. Assuming I have correctly understood the intent of those rules (which in a real world scenario I would seek to clarify), we can safely ignore the first requirement so long as the second is correctly enforced.</p>
<h2 id="evolving-design">Evolving Design</h2>
<p>Let’s start with the most obvious, albeit naive approach of simply using a random sequence. Be it <code class="highlighter-rouge">Random.Next</code> or one of the cryptographic sequences available in .Net, we can easily get a random sequence, and let’s assume for now we have a function that can select a resource from our pool of resources given part of this sequence. What are the problems with such a design?</p>
<p>Let’s label our resources <code class="highlighter-rouge">A</code> to <code class="highlighter-rouge">J</code> to give us 10 unique resources. Let’s also assume the random sequence we generate provides integers, which allows as to easily zero-index our resources (<code class="highlighter-rouge">0</code> = <code class="highlighter-rouge">A</code>, <code class="highlighter-rouge">1</code> = <code class="highlighter-rouge">B</code>, <code class="highlighter-rouge">2</code> = <code class="highlighter-rouge">C</code>, … <code class="highlighter-rouge">9</code> = <code class="highlighter-rouge">J</code>).</p>
<p>In order to better communicate intent, let’s call the first day of a 2 week period <em><code class="highlighter-rouge">d</code></em>, the second day of the week being <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+1</code>, the third day <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+2</code> and so forth. We also have two shifts per day, so let’s call the AM shift <em><code class="highlighter-rouge">a</code></em> and the PM shift <em><code class="highlighter-rouge">b</code></em> (note casing difference to our resources so as not to confuse them). Therefore, the aftenoon shift of the 5th day in a 2 week period we’ll describe as <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+4</code><em><code class="highlighter-rouge">b</code></em>. If that doesn’t make sense to you yet, hopefully it’ll become clear as we work our way through this.</p>
<p>The first obvious problem in my mind is that the sequence <code class="highlighter-rouge">0, 0, 0, 0, 0, 0, 0, ...</code> is essentially just as likely as any other random sequence you can think of. I recall some years ago watching a documentary that showed that a subset of people have been choosing consecutive sequences (<code class="highlighter-rouge">1, 2, 3, 4...</code>) in the national lottery for as long as it’s been around; it’s just as likely as any other seemingly random sequence.</p>
<p>We can easily side-skirt this issue by running the business rules when a random number is selected. If the random number will cause one of our rules to break, simply ignore it and take the next number in the sequence. However, in order to test the second business rule (because we can ignore the first) we need to know who has been allocated to the shifts yesterday and/or tomorrow, and in order to achieve the third business rule, we need to consider an entire 2 week period.</p>
<p>If we assume our implementation iterated through the 2 week period, starting at <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+0</code><em><code class="highlighter-rouge">a</code></em> (or <em><code class="highlighter-rouge">da</code></em> for short), moving on to <em><code class="highlighter-rouge">db</code></em>, then <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+1</code><em><code class="highlighter-rouge">a</code></em> and so forth, we can easily track the days allocated so far, and enforce rule 2 by simply skipping random numbers if the resource derived from the number has appeared in the last 2 days. In doing so however, it’s perfectly plausible that we paint ourselves into a corner:</p>
<table>
<thead>
<tr>
<th style="text-align: left"><strong>Week 1:</strong></th>
<th style="text-align: center"><em><code class="highlighter-rouge">da</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">db</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+1</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+1</code><em><code class="highlighter-rouge">b</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+2</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+2</code><em><code class="highlighter-rouge">b</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+3</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+3</code><em><code class="highlighter-rouge">b</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+4</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+4</code><em><code class="highlighter-rouge">b</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+5</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+5</code><em><code class="highlighter-rouge">b</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+6</code><em><code class="highlighter-rouge">a</code></em></th>
<th style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+6</code><em><code class="highlighter-rouge">b</code></em></th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left"> </td>
<td style="text-align: center">A</td>
<td style="text-align: center">B</td>
<td style="text-align: center">C</td>
<td style="text-align: center">D</td>
<td style="text-align: center">A</td>
<td style="text-align: center">B</td>
<td style="text-align: center">C</td>
<td style="text-align: center">D</td>
<td style="text-align: center">A</td>
<td style="text-align: center">B</td>
<td style="text-align: center">C</td>
<td style="text-align: center">D</td>
<td style="text-align: center">A</td>
<td style="text-align: center">B</td>
</tr>
<tr>
<td style="text-align: left"><strong>Week 2:</strong></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+7</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+7</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+8</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+8</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+9</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+9</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+10</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+10</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+11</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+11</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+12</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+12</code><em><code class="highlighter-rouge">b</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+13</code><em><code class="highlighter-rouge">a</code></em></td>
<td style="text-align: center"><em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">+13</code><em><code class="highlighter-rouge">b</code></em></td>
</tr>
<tr>
<td style="text-align: left"> </td>
<td style="text-align: center">C</td>
<td style="text-align: center">D</td>
<td style="text-align: center">A</td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
<td style="text-align: center"> </td>
</tr>
</tbody>
</table>
<p>As you can hopefully see here, I’ve allocated the first 4 resources (<code class="highlighter-rouge">A</code> to <code class="highlighter-rouge">D</code>) sequentially for the first 8 and a half days, leaving 11 slots as yet unallocated. At this stage though, no matter what comes next, it is no longer possible to achieve rule 3. That is because I have 6 resources (<code class="highlighter-rouge">E</code> to <code class="highlighter-rouge">J</code>) that must be allocated to 2 slots each within this period, which would require 12 of my 11 remaining slots.</p>
<p>You could argue that the likelihood of having the <code class="highlighter-rouge">A</code> to <code class="highlighter-rouge">D</code> sequencing is very unlikely, and I agree with you. However, I’ve simply used this sequencing to emphasise the issue. You could randomly sequence 9 of the 10 resources for 24 of the total 28 slots in a 2 week period, and still fall foul of the business rules because you would not be able to allocate the remaining resource into the 2 slots that rule 3 mandates without breaking rule 2.</p>
<p>This is not the only problem either, what about the slots <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">a</code></em> and <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">b</code></em> (that is, the day <em>before</em> this 2 week period started). It’s not even visible to our chosen scope, but if those two slots contained resources <code class="highlighter-rouge">A</code> or <code class="highlighter-rouge">B</code>, we’d have broken rule 2 without even being able to see it!</p>
<p><img src="../images/funny-concentrationcat.jpg" alt="funny concentrating cat" /></p>
<h2 id="changing-tack">Changing Tack</h2>
<p>The final paragraph in the previous section highlights a pretty significant issue with regards to design of this system. One could argue that we simple add <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">a</code></em> and <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">b</code></em> to our scopes in order to circumvent the possibility of accidentally breaking rule 2 on <em><code class="highlighter-rouge">da</code></em> and <em><code class="highlighter-rouge">db</code></em>, but what if <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">a</code></em> or <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">b</code></em> were forced to skip one or more random numbers due to similar constraints placed on them by either rule 3, or rule 2 due to allocations in <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-2</code><em><code class="highlighter-rouge">a</code></em> and <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-2</code><em><code class="highlighter-rouge">b</code></em>? If you were to follow that line of reasoning, you would need <strong>every</strong> day in history to know for sure that you weren’t going to break rules.</p>
<p>The technical test did mention that I could use any <em>ORM</em> I wanted, so perhaps that is the intent? When the system starts, I could let the application sit and churn for hours calculating rotas for all of time. The thought of doing such a thing actually repulses me somewhat, and fortunately I know of a mechanism that, combined with a little creativity, should completely circumvent that necessity.</p>
<p>For anyone that plays computer games, you’ll have come across seeded randoms, typically in world generation algorithms like those used by <em>Minecraft</em>. The games are capable of generating (either upfront or procedurally, as is the case for <em>Minecraft</em>) a random environment based on a seed value. The world is for all intents and purposes completely random, but anyone using the same seed value will also generate the exact same random world.</p>
<p>It’s perhaps not very well known, but even .Net’s <a href="https://msdn.microsoft.com/en-us/library/ctssatww(v=vs.110).aspx"><code class="highlighter-rouge">Random</code> class</a> can take an <code class="highlighter-rouge">int</code> during construction to act as a seed. This is useful, because if employed correctly, it would allow our random rota code to examine any days outside of our 2 week scope, such as the aforementioned <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">a</code></em> and <em><code class="highlighter-rouge">d</code></em><code class="highlighter-rouge">-1</code><em><code class="highlighter-rouge">b</code></em> slots.</p>
<p>You could still face issues like those described earlier if numbers in the random sequence had to be skipped due to constraints coming from other days outside of our scope, but not if we use multiple phases and introduce <em>anchor</em> days.</p>
<p>Imagine the algorithm works as follows:</p>
<ol>
<li>Instantiate the seeded random using the epoch of our anchor day within the 2 week scope (whether that anchor day is <em><code class="highlighter-rouge">d</code></em> or <em><code class="highlighter-rouge">d13</code></em> is up to you, but the start and end of the 2 week scope make the best candidates in my opinion).</li>
<li>Populate the anchor day in accordance with rule 2. That is, the first slot is guaranteed to be filled by the first number in the random sequence because all other slots are empty, then iterate that sequence until a <em>different</em> resource is selected for the second slot in that day.</li>
<li>Instantiate a second seeded random using the epoch of the period either before or after our scope (depending on which anchor you selected), and calculate the slot allocations that would have been used for that period. We know they follow the sames rules as our anchor day and because they are allocated first and foremost, they only need to account for rule 2, as ours did.</li>
<li>Next, focus on rule 3 by iterating the resources (taking into account the 2 slots already allocated to our anchor day). Randomly assign each resource to a random slot within the period, taking into account what we know of the anchor day in the adjacent period, and skipping the sequence if rule 2 would be broken.</li>
<li>Once all resources have a 2 slots each within scope ensuring our compliance with rule 3, continue allocating randomly however you want, taking into account rule 2 and the adjacent anchor.</li>
</ol>
<p>This has the benefit of never needing to store anything. We can generate the rota for any period in the past, present, or future, and be confident that we’ll meet our requirements. This part of the overall implementation appears to be immune to requirement changes, such as needing to change the number of resources, length of period, minimum shift count, and minimum allowed adjacency to shifts. Sure, we’ll need to make these aspects extensible/configurable, but the algorithm should continue to function as intended if those changes are requested at a later date.</p>
<h2 id="tidying-up">Tidying up</h2>
<p>The final piece of the puzzle is to align our periods to ensure we don’t accidentally have 2 week periods overlapping with one another. This, again, is tad trickier than it might seem on first glance. The number of weeks in a year vary, and even the way we number those weeks varies. The accepted format is typically <em>ISO 8601</em>, however, the .Net implementation doesn’t quite agree with that ISO, as shown <a href="https://blogs.msdn.microsoft.com/shawnste/2006/01/24/iso-8601-week-of-year-format-in-microsoft-net/">here</a>.</p>
<p>We can massively simplify this though. As mentioned above, we have to consider it possible for the business requirements to change the length of the period we’re working with. Right now it is 2 weeks, but that doesn’t necessarily tie us to operating in weeks (what if the business wants to change the period to 8 and a half days?).</p>
<p>To alleviate this issue, let’s take the epoch (which as of .Net 4.6 and .NetStandard 1.3, we can easily pull from <a href="https://msdn.microsoft.com/en-us/library/system.datetimeoffset.tounixtimeseconds(v=vs.110).aspx"><code class="highlighter-rouge">DateTimeOffset.ToUnixTimeSeconds()</code></a>) and perform a modulo operation with the epoch as <em>dividend</em> and period length in seconds as <em>divisor</em>. This will give us the remainder, which can be deducted from the epoch used to identify an anchor. Adding the period length to that anchor gives us a second anchor, and doing so again can get a third and so forth if required.</p>
<p>Regardless of what period a user might ask for, we can populate rotas aligned to those anchors in order to provide the period requested. For example, if the user asks to see the rotas for a 3 week period (continuing to use 2 week rota periods), we can confidently generate 2 week periods behind the scenes and then limit our results to the period the user requested.</p>
<p>As with the random allocation algorithm, this should also be immune to the requirement changes we identified above.</p>
<h2 id="known-issues">Known Issues</h2>
<p>Whilst our algorithm will continue to function as intended regardless of the requirement changes identified, if any of our parameters are modified, that algorithm will no longer align periods as it did before, and/or no longer populate the periods in the same manner. Therefore, the application will be unable to fulfill it’s requirements because it is unable to predict the location and/or allocations of the anchors used prior to the parameters having been modified, opening us up to the same problems we identified before having introduced those anchors.</p>
<p>I think, for now, that would be an acceptable caveat for the design (given I don’t have an actual product owner available to verify that decision in the context of a technical test). We could alleviate the issue by storing any periods we allocate, but I’ll leave that as an optional extra I’ll skip for the sake of brevity.</p>I recently went back out to market in search of a new role and was quickly put on to a really exciting company. The outcome to my application to this company is not yet determined, but part of the process is to complete what appears at first glance to be a simple technical test. However, having given the test a bit of thought throughout the course of the day, it’s actually a lot more complicated than it first appears. In fact, I spent quite a while completely stumped as to a satisfactory design and implementation. I’m not sure at this stage whether this was intentional, some sneaky derivative of some known difficult problem that, having never completed college or university I’m unaware of, or if I was just over-thinking matters.Agile Security2017-09-05T00:00:00+00:002017-09-05T00:00:00+00:00http://blog.devbot.net/agile-security<p>Over the past several months, the company I work for has been creating a new product which ties into <em>“Critical UK Infrastructure”</em>. As you might imagine, the security obligations that come with such integration are… considerable. I’ve always thought myself pretty savvy when it comes to securing the software I write, with sufficient networking, operating system, and web knowledge to at least apply <em>above average</em> controls to ensure the protection of the data that the systems I work with handle. I’ll freely admit that my ability to document and convey the controls, risks, systems, have <em>not</em> been sufficient to pass the arduous government driven security assessments faced without a fair degree of re-education. As part of that re-education though, I’ve definitely picked up and put into place practices that I’d continue in the future, even working on smaller projects or those without any obligations beyond the layman’s honour to protect ones clients and customers.</p>
<p><img src="https://img.memecdn.com/security_o_5146659.webp" alt="funny security" /></p>
<p style="font-size: 12px; text-align: center;"><em>Image from <a href="https://www.memecenter.com/fun/5146659/security">MemeCenter</a></em></p>
<p>In this article, I want to describe some of the practices I now use, and will continue to use in future agile projects. My intent isn’t to describe to you how Enterprise scale software should be written or how to protect against SQL injection; instead I want to show the average developer what they should be considering, and how this can work seamlessly with your existing methodologies.</p>
<blockquote>
<p>I recently gave a <a href="https://youtu.be/45F1rjzvkuA">30 minute talk</a> on this subject at <a href="https://meet.dnbmth.io">.Net Bournemouth</a> - not all aspects of this article are covered by the talk (I had to cut a lot to fit it into 30 minutes), but feel free to check it out!</p>
</blockquote>
<h2 id="threats">Threats</h2>
<p>To begin to understand why you should be spending time reading this article and improving your security practices, take a look at this <a href="https://map.norsecorp.com/#/">impressive visualisation of live attacks</a> from <a href="http://www.norse-corp.com/">Norse Corp</a>. Honestly, open it in a new a tab and check it out for a minute. I’ll wait…</p>
<p>Ok, so that map just shows sensors being scanned and provoked, but no harm done. Right?</p>
<p>Wrong. Check out the number of breaches and the insane volume of data already breached on <a href="https://www.troyhunt.com/">Troy Hunt’s</a> site, <a href="https://haveibeenpwned.com/PwnedWebsites">Have I Been Pwned</a>. At time of writing, it’s a little under 5 <strong>BILLION</strong> accounts. And these are just the ones we know about.</p>
<p>Whilst each of those breaches will have specific circumstances and reasoning, I’d argue that our industry as a whole is culpable of, at best, <em>gross negligence</em>, and in certain cases, probably worse. The data our systems process and hold, the details of our customers and clients is the <em>personal property</em> of those individuals. And we are all too often <em>facilitating</em> the theft of that property, be it through incompetence, inadequate education, or negligence.</p>
<p>But it’s not (I’d hope) because we <em>want</em> our customers to suffer at the hands of these would-be attackers, in the same way none of us would want to be subject to (more) spam, identity fraud, and any other number of malicious electronic activities made possible by the loss of our details. So what more can we be doing to prevent it?</p>
<h2 id="assets">Assets</h2>
<p>To understand what damage can be done and what it is we should be protecting, we need to understand our system’s <em>Assets</em>. These can include a whole lot more than you might first think - do you have keys to your building? Passes to get into the car park? A laptop that you take home with you?</p>
<p>In our quest to defend our systems and data, we should also be protecting physical assets that would otherwise accomodate an attack. We’re often responsible for much more than just the lines of code that we write.</p>
<h3 id="secrets">Secrets</h3>
<p>It’s not just direct access to our Customer Databases we need to be thinking about. How are your system credentials, user passwords, API keys stored? How are they generated and communicated? Seems obvious, right? But check out how many commits on github are named <a href="https://github.com/search?l=&q=removed+password&ref=advsearch&type=Commits&utf8=%E2%9C%93">“removed password”</a>. More than 300,000 passwords just broadcast to the world.</p>
<p>Do you think each of those authors owned up to their mistake? Did each of them change the passwords to the infrastructure they’d just handed over to attackers? What would you have done?</p>
<p>There are numerous mechanisms and ample advice available to help protect these secrets, such as Microsoft’s <a href="https://docs.microsoft.com/en-us/aspnet/core/security/app-secrets">Secret Manager</a>, Octopus’ <a href="https://octopus.com/docs/deploying-applications/variables/sensitive-variables">Sensitive Variables</a>, App Veyor’s <a href="https://www.appveyor.com/docs/build-configuration/#secure-variables">Secure Variables</a>, etc. Every applicable platform seems to have mechanisms and guidance to help you secure your secrets so use them. Always. It doesn’t matter how valuable you think a given secret is, get into the practice of always securing your secrets so you’re never liable to slip up and drop a password or key that <em>does</em> have value to attackers.</p>
<h3 id="records">Records</h3>
<p>The secrets themselves are rarely likely to damage your business and customers directly, instead being used to get access to data you were trying to protect. It’s easy to see the breach of customer data as something abstract, just a number or the impact of someone we know little of or care little for directly. Even if that is the case, we have to understand and implement <em>defense-in-depth</em>. If an attacker is able to compromise your web server or other fringe infrastructure (proxies, firewalls, load balancers, etc), gaining access to an internal network, what more might they be able to do thereafter?</p>
<p>Financial, legal, internal documentation for your company could potentially be used to blackmail the company or simply disclosed at severe cost of reputation. It could also effect you, <em>personally</em>. What if your HR data is leaked? Information on disciplinaries, protected characteristics, your bank details and address, information for your next of kin, inflammatory remarks sent between colleagues…</p>
<p>It’s not just one more thing you need to be juggling day to day to keep your bosses happy, it’s not just your professional reputation at stake, consider what disclosure of this data could do to <strong>your career</strong>, how it could effect those <strong>you care about</strong>, the impact it could have upon <strong>your life</strong>.</p>
<p>The sheer mass of existing breaches, the balancing act of so many requirements, the all too common detachment of developer and end user can easily numb us to the impact attacks might have. But we need to live and breath security whilst we continue to deliver value.</p>
<h2 id="plan">Plan</h2>
<p>So now that we know the realworld impacts loss and destruction of data can have upon us and those we care about, hopefully you want to do something about it? This should be a moral obligation upon all of us, but knowing how it might hamper us personally will hopefully help us emphathise with those we’re charged with protecting.</p>
<p>Maybe you’re already using an ORM to protect you from SQL injection, maybe you’ve read up on XSS attacks, so what more can you be doing? What is the plan?</p>
<p><img src="../images/security-plan.png" alt="security plan" /></p>
<p>The rest of this article will walk you through how to implement threat modelling, how to conduct threat assessments, and how to integrate these practices snugly with your existing agile processes. There’s no point undertaking this excercise just once or giving up half way through, we need these practices to be as convenient as possible if they’re to have an ongoing, valuable impact.</p>
<blockquote>
<p>If you’re not following a templated/typical agile process such as Scrum, don’t fear. So long as you follow some form of <em>plan -> do -> review</em> process, I’m sure you can make this work.</p>
</blockquote>
<h2 id="model">Model</h2>
<p>If you were to ask any two developers for a given product what it is that constitutes that product’s application(s), you’d likely get very different answers. This isn’t necessarily a bad thing, but we do need to find a way to describe our system in such a way that we can examine the security aspects. This practice is often described as a part of <a href="https://www.owasp.org/index.php/Threat_Risk_Modeling">Threat Modeling</a>.</p>
<p>Unless you’re about to start a new green field project, decomposing your application is typically one of the largest investments you’ll need to make with regards to integrating improved security practices into your workflow. Don’t be disheartened though, it probably won’t take as long as you expect, even for larger applications (and the larger your application, the more <em>necessary</em> this work probably is). We’ll start by determining what exactly your application is.</p>
<h3 id="scope">Scope</h3>
<p>The first step is work out which <em>components</em> make up your application. Find a whiteboard or a nice big blank piece of paper and work with your team to break out the system into it’s individual <em>bits</em>. In an era of microservices and serverless functions, there might well be several moving parts, but it shouldn’t take long to identify each component.</p>
<p>I recommend doing this in one or more diagrams, visualise your application in all it’s glory. This artchitectural overview should be fairly easy to keep up to date going forward, and will be very useful examining the next aspects of your application.</p>
<blockquote style="font-size: 12px; text-align: center;">
<p>If you haven’t undertaken an excercise like this before, what you’re aiming for is a drawing a bit like this:</p>
<p><img src="../images/security-components.png" alt="application components" />
<em><a href="https://www.4com.co.uk/">4Com Plc ©</a></em></p>
</blockquote>
<blockquote>
<p>It doesn’t need to make sense to the outside world necessarily, so long as your team agree that it encompasses everything of note, and it makes sense to <em>them</em>.</p>
</blockquote>
<p>You may find that the application integrates with other products and applications. This is where you need to work out the <em>scope</em> of your application. You need to draw a line between your application and the others, which means determining on which side of that line each component should land. If your team is responsible for both applications, ensure someone is playing <em>devil’s advocate</em>, making sure that you’re not unduly expanding the scope of another product in order to make life simpler for the application you’re modelling. Similarly, don’t absorb other product’s components, it will only hamper you further down the line.</p>
<h3 id="decomposition">Decomposition</h3>
<p>Now that we’ve defined what our application actually is, we have a scope to work to. This scope can always be changed later as the application evolves, but will keep us constrained for now and hopefully prevent us going off at tangents.</p>
<p>Like I said, this initial investment can be a bit tedious for brownfield products. You’ve probably already generated a couple new documents to maintain, and you’re watching document-hell creep over the horizon. Just remember, keep it simple. These lists and pictures simply need to make sense to you and your team. Contrary to my blog posts, you don’t need lengthy descriptions and to be working on the basis that newcomers will learn <em>everything</em> from these documents.</p>
<p>Depending on the complexity of your application, it may now be worthwhile breaking your system down further.</p>
<h4 id="application-assets">Application Assets</h4>
<p>Identify the assets mentioned earlier like passwords, API keys, configuration data, connection strings, data input by users or received from third parties, etc. Simply start a list and keep adding to it until you can’t think of any more, updating it from time to time as features are added or you think of something you missed. As we break down the system further, you’ll inevitably think of more. That’s fine, just keep coming back to this list and adding to it.</p>
<p>I recommend considering how <em>sensitive</em> each of your assets are, how much value they represent and how much scrutiny should be invested in keeping each asset secure. Coming up with some simple tiers to categorise each of your assets, or following a more formal <em>Information Classification Scheme</em> (Public, Restricted, Confidential, etc) will help later when it comes to prioritising the extent of measures justifiable in securing these assets.</p>
<p>How much information you want to hold on these assets is up to you, but I recommend starting simple. The harder you make these lists, the less likely people will keep them up to date! <em>Name</em>, <em>Description</em> where necessary, and <em>Classification</em> make a nice starting point which you can expand upon later as required.</p>
<p>Depending on the size of the system you’re trying to decompose, it may make life easier to categorise your assets (different sheets in an Excel workbook or separate markdown tables, whatever your flavour). This will help you track extra relevant information should you feel the need. For example, physical assets (switches, monitors, USB sticks.) may warrant extra columns for <em>make</em>, <em>model</em>, <em>serial number</em>, <em>firmware version</em>, and so forth, that wouldn’t make sense when describing code repositories and passwords.</p>
<h4 id="external-dependencies">External Dependencies</h4>
<p>Consider the fully deployed version of the system infront of you. What technologies are involved in the delivery of your software? Throw Teamcity, TFS, Octopus, Docker, Kubernetes, and any other cool thing you use on this list. They may not be <em>very</em> important when it comes to securing your application, but they’re probably worthy of consideration.</p>
<p>How about infrastructure such as load balancers, proxies, routers? What are you depending on in production? Do you have a log server? Is your web application sat behind IIS? Include all of these too! Whilst you may not be directly responsible for securing these dependencies, they are a part of your threat model. For example, if a hacker can telnet into your proxy server and shut it down then your system becomes unavailable, or perhaps they can re-route your customer traffic to a server of <em>their</em> choosing. All that work you did securing your application just to have an attacker route all traffic off to a <a href="https://en.wikipedia.org/wiki/Phishing">phishing</a> site.</p>
<p>For now, simply listing the names (maybe a brief description) of these dependencies should be more than sufficient. The idea is simply to <em>not</em> forget them when we’re assessing for threats!</p>
<h4 id="entry-points">Entry Points</h4>
<p><img src="https://www.cubebreaker.com/wp-content/uploads/2014/10/useless-things-feat.jpg" alt="pointless gate" /></p>
<p>This is where we start getting into the meat of the matter. What are the points of access into your system? This isn’t as simple as it first seems. Can you remote desktop to one of the servers in your system? Can you telnet into the load balancer? Can you login to your AWS portal? Can you access server storage across your internal network? Like all the sections above, you need to be thinking outside of the box.</p>
<p>So many attacks succeed because of our reliance on single points of failure. We can’t ignore a vulnerability just because it isn’t immediately accessible. If the thing infront of it is compromised, what can we do to defend? Many of you will have played some form of <em>tower defense</em> game; would you win those games with just a single tower at an entry point?</p>
<p>As with identifying assets, it’ll probably take a couple passes to identify the more obscure entry points, so just keep a list to hand and update it as you go.</p>
<p>In addition to the more obscure entry points are the items you’re likely much more familiar with - what pages and API calls are there? What message buses and file inputs does the system read from? What web hooks are you subscribed to? Again, add all of these to the list (or additional lists if thats easier for you to manage).</p>
<p>For every one of these endpoints, you’re going to want a name and probably a description to help identify them. You’ll also need to add what <em>level of trust</em> is required to utilise each end point. For example, what happens when an <em>Anonymous Web User</em> navigates to your search page? Or when an <em>Authenticated User</em> from the Customer Support team attempts to access confidential HR data? The simplest way we found to track this information is list the users, service accounts, and/or roles that <em>should</em> have access to an entry point, whether it is implemented yet or not.</p>
<h4 id="trust-levels">Trust Levels</h4>
<p>If you’ve already worked through your entry points and identified which users, service accounts, and roles should have access to various entries, this should be really easy. Simply list all of the roles, or get fancy and hookup some kind of <code class="highlighter-rouge">VLOOKUP</code> in Excel, whatever works for you.</p>
<p>As mentioned previously, you may not yet have implemented the necessary controls to restrict access in accordance with these roles; identifying them is the crucial aspect though. Little more than <em>Name</em> and <em>Description</em> should be required here.</p>
<h4 id="data-flows">Data Flows</h4>
<p>The final piece of modelling your application is to understand and where necessary, document your data flows. These flows rely upon and consolidate all of the information you’ve gathered up to now.</p>
<p>The intent is to show how data (<a href="#application-assets">Assets</a>) enter the system (<a href="#entry-points">Entry Points</a>), which components (within <a href="#scope">Scope</a>) process, store, or transfer that data. Which users should access or provide that data (<a href="#trust-levels">Trust Levels</a>), which third parties (<a href="#external-dependencies">External Dependencies</a>) receive or provide data?</p>
<p>If you want to follow guidance from <a href="https://www.owasp.org">OWASP</a>, you might end up with a diagram like this:</p>
<p><a href="https://www.owasp.org/index.php/Application_Threat_Modeling#Data_Flow_Diagrams"><img src="https://www.owasp.org/images/0/00/Data_flow1.jpg" alt="owasp data flow" /></a></p>
<p>Or if you want to try and keep things simple, go back to the whiteboard and scribble down something a bit more rough and ready. The object is not to draw the best diagram, it’s to model your application. If that means some scribbles in a notebook that you take a picture of and throw into a docs folder in your repo, that’s absolutely fine.</p>
<p>If you’re able to visualise these interactions, it’s going to make your job of assessing them <em>that</em> much easier.</p>
<h2 id="assess">Assess</h2>
<p>You now have all the information you require to assess your system. You may have missed pieces, but as has been mentioned, simply go back to the relevant lists and add information as it comes to mind. As you’ve been going through those lists, you’ve probably already spotted a couple places that could do with some improvement.</p>
<p>There are several ways to go about assessing your system, and several aspects of that system that need assessment. I’ll list below a couple mechanisms that I have used and recommend to others, but this is not an exhaustive list. Any and all thought towards the security of your system should be welcome, and I’d love to hear any thoughts you might have on other ways to analyse applications.</p>
<h3 id="access-control">Access Control</h3>
<p>###</p>Over the past several months, the company I work for has been creating a new product which ties into “Critical UK Infrastructure”. As you might imagine, the security obligations that come with such integration are… considerable. I’ve always thought myself pretty savvy when it comes to securing the software I write, with sufficient networking, operating system, and web knowledge to at least apply above average controls to ensure the protection of the data that the systems I work with handle. I’ll freely admit that my ability to document and convey the controls, risks, systems, have not been sufficient to pass the arduous government driven security assessments faced without a fair degree of re-education. As part of that re-education though, I’ve definitely picked up and put into place practices that I’d continue in the future, even working on smaller projects or those without any obligations beyond the layman’s honour to protect ones clients and customers.Logging - Tips and Tricks2017-01-15T00:00:00+00:002017-01-15T00:00:00+00:00http://blog.devbot.net/logging-practices<p>I tweeted a thank you to the guys over at <a href="https://getseq.net/">Seq</a> recently because <em>Seq</em> was instrumental in finding and eliminating a very difficult issue we came across. But technology choice alone isn’t enough, so I’ve decided to blog about some of the logging practices we tend to follow as a rule, and some specific practices that we used in a particularly complex application we’ve been developing over the last couple of months.</p>
<blockquote class="twitter-tweet" data-partner="tweetdeck"><p lang="en" dir="ltr">Logging best practices and <a href="https://twitter.com/getseq_net">@getseq_net</a> have been an absolute god-send for tracking down tricky problems. Much love. <a href="https://t.co/6fubnhSnOo">pic.twitter.com/6fubnhSnOo</a></p>— Tommy Long (@Smudge202) <a href="https://twitter.com/Smudge202/status/816749326642847749">January 4, 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="contents">Contents</h2>
<ul id="markdown-toc">
<li><a href="#contents" id="markdown-toc-contents">Contents</a></li>
<li><a href="#foreword" id="markdown-toc-foreword">Foreword</a> <ul>
<li><a href="#service-collection-extensions" id="markdown-toc-service-collection-extensions">Service (Collection) Extensions</a></li>
</ul>
</li>
<li><a href="#abstractions" id="markdown-toc-abstractions">Abstractions</a></li>
<li><a href="#sinks-basic" id="markdown-toc-sinks-basic">Sinks (Basic)</a></li>
<li><a href="#what-to-log" id="markdown-toc-what-to-log">What to Log</a> <ul>
<li><a href="#exceptions" id="markdown-toc-exceptions">Exceptions</a></li>
<li><a href="#decisions" id="markdown-toc-decisions">Decisions</a></li>
<li><a href="#state" id="markdown-toc-state">State</a></li>
<li><a href="#correlation" id="markdown-toc-correlation">Correlation</a></li>
<li><a href="#performance" id="markdown-toc-performance">Performance</a></li>
</ul>
</li>
<li><a href="#what-level-to-log" id="markdown-toc-what-level-to-log">What Level to Log</a> <ul>
<li><a href="#log-levels" id="markdown-toc-log-levels">Log Levels</a></li>
<li><a href="#log-spam" id="markdown-toc-log-spam">Log Spam</a></li>
</ul>
</li>
<li><a href="#practices" id="markdown-toc-practices">Practices</a> <ul>
<li><a href="#interpolation" id="markdown-toc-interpolation">Interpolation</a></li>
<li><a href="#disposable-tracking" id="markdown-toc-disposable-tracking">Disposable Tracking</a></li>
</ul>
</li>
<li><a href="#sinks-advanced" id="markdown-toc-sinks-advanced">Sinks (Advanced)</a> <ul>
<li><a href="#seq" id="markdown-toc-seq">Seq</a></li>
</ul>
</li>
<li><a href="#summary" id="markdown-toc-summary">Summary</a></li>
</ul>
<h2 id="foreword">Foreword</h2>
<p>This is going to be a lengthy post, and several of the practices shown are circumstantial, subjective, situational, and so forth. Add these practices to your developer toolbelt, and decide for yourself when they best fit your needs.</p>
<p>The system we’ve been working on is a series of commercial applications subject to standard IP clauses, so I’ve included as much code as I can in this post. Wherever possible the code is sans abstraction and obfuscation, but I won’t be able to provide the applications themselves as samples.</p>
<p>The code is primarily contained within 15 .Net Core libraries targeting the full .Net Framework (4.5.2) due to dependency requirements. 4 of those libraries are what we call <em>application hosts</em>, containing no business logic, designed to host a selection of the libraries as a .Net Core Console Application which can be deployed inside <a href="https://www.docker.com/">Docker</a> containers, as a Windows Service, or in Azure. (This is a practice that our company tends to follow and has a collection of composition helpers for so that the applications can be ran anywhere, leaving deployment considerations to our IT department.)</p>
<p>In addition to the above, we have a considerable amount of heritage code in other solutions which is NuGet packaged and deployed by TFS/Octopus, which is then pulled down by one of our microservices at runtime and installed/executed within a separate <code class="highlighter-rouge">AppDomain</code>. As you might imagine, with this many moving parts, logging is essential.</p>
<h3 id="service-collection-extensions">Service (Collection) Extensions</h3>
<p>Our company has adopted the <code class="highlighter-rouge">IServiceCollection</code> extension method approach of building up our composition roots. That is, our <em>application host</em> will contain code similar to the following:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="k">public</span> <span class="k">static</span> <span class="kt">int</span> <span class="nf">Main</span><span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">services</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServiceCollection</span><span class="p">()</span>
<span class="p">.</span><span class="nf">AddServicesFromOurClassLibrary</span><span class="p">();</span>
<span class="n">services</span><span class="p">.</span><span class="nf">BuildServiceProvider</span><span class="p">()</span>
<span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">SomeService</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">Run</span><span class="p">();</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p><em>NB: Extremely simplified for sake of brevity.</em></p>
<p>In turn, our class libraries expose their services as follows:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection.Extensions</span><span class="p">;</span>
<span class="k">public</span> <span class="k">static</span> <span class="n">IServiceCollection</span> <span class="nf">AddServicesFromOurClassLibrary</span><span class="p">(</span><span class="k">this</span> <span class="n">IServiceCollection</span> <span class="n">services</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">services</span><span class="p">.</span><span class="n">TryAddSingleton</span><span class="p"><</span><span class="n">Foo</span><span class="p">,</span> <span class="n">Bar</span><span class="p">>();</span>
<span class="k">return</span> <span class="n">services</span><span class="p">;</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Whilst it’s up to you how you manage the composition of your application, this is based on the practice used by ASP.Net Core and is the assumption throughout the rest of this article.</p>
<h2 id="abstractions">Abstractions</h2>
<p>First and foremost, let’s get our abstractions installed. <em>All</em> of our class libraries depend upon the <a href="https://nuget.org/packages/microsoft.extensions.logging.abstractions"><code class="highlighter-rouge">Microsoft.Extensions.Logging.Abstractions</code></a> NuGet Package. The package is lightweight as you would expect of any series of abstractions. Whilst there are other abstractions available, some of which we inevitably need to integrate with through façades, I’ve found this one of the most well adopted logging abstractions, especially given the recent flurry of activity surrounding .Net Core.</p>
<p>Once installed, don’t forget to add <code class="highlighter-rouge">services.AddLogging()</code> to your service extensions. After which, any class can rely upon <code class="highlighter-rouge">ILogger<TCategory></code> as a dependency, giving us an easy to use logging abstraction.</p>
<p><em>NB: This step alone will not <strong>output</strong> any logs.</em></p>
<h2 id="sinks-basic">Sinks (Basic)</h2>
<p>With the above in place, your application hosts become responsible for hooking up log outputs/sinks so that you’re logs actually go somewhere. I recommend the following as a bare minimum:</p>
<ul>
<li><a href="https://nuget.org/packages/microsoft.extensions.logging"><code class="highlighter-rouge">Microsoft.Extensions.Logging</code></a></li>
<li><a href="https://nuget.org/packages/microsoft.extensions.logging.debug"><code class="highlighter-rouge">Microsoft.Extensions.Logging.Debug</code></a></li>
<li><a href="https://nuget.org/packages/microsoft.extensions.logging.console"><code class="highlighter-rouge">Microsoft.Extensions.Logging.Console</code></a></li>
</ul>
<p>Personally, I prefer to swap out the <code class="highlighter-rouge">Microsoft.Extensions.Logging.Console</code> for the following, which will give me coloured console output:</p>
<ul>
<li><a href="https://nuget.org/packages/serilog.extensions.logging"><code class="highlighter-rouge">Serilog.Extensions.Logging</code></a></li>
<li><a href="https://nuget.org/packages/serilog.sinks.literate"><code class="highlighter-rouge">Serilog.Sinks.Literate</code></a></li>
</ul>
<p>By adding the <code class="highlighter-rouge">Serilog.Extensions.Logging</code> package, you open up your application to dozens of high quality outputs via <a href="https://github.com/serilog/serilog/wiki/Provided-Sinks">Serilog Sinks</a>. For example, if you’re working on an application that does not have a Console, such as a Windows service, you might consider adding:</p>
<ul>
<li><a href="https://nuget.org/packages/serilog.sinks.rollingfile"><code class="highlighter-rouge">Serilog.Sinks.RollingFile</code></a></li>
</ul>
<p>We can now adapt the application host code shown above to something like:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
</pre></td><td class="rouge-code"><pre><span class="k">using</span> <span class="nn">Microsoft.Extensions.DependencyInjection</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Microsoft.Extensions.Logging</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Serilog</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">Serilog.Events</span><span class="p">;</span>
<span class="k">public</span> <span class="k">static</span> <span class="kt">int</span> <span class="nf">Main</span><span class="p">(</span><span class="kt">string</span><span class="p">[]</span> <span class="n">args</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">services</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">ServiceCollection</span><span class="p">()</span>
<span class="p">.</span><span class="nf">AddServicesFromOurClassLibrary</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">provider</span> <span class="p">=</span> <span class="n">services</span><span class="p">.</span><span class="nf">BuildServiceProvider</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">loggerFactory</span> <span class="p">=</span> <span class="n">provider</span><span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">ILoggerFactory</span><span class="p">>();</span>
<span class="n">loggerFactory</span><span class="p">.</span><span class="nf">AddDebug</span><span class="p">(</span><span class="n">LogLevel</span><span class="p">.</span><span class="n">Trace</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">serilogConfiguration</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">LoggerConfiguration</span><span class="p">()</span>
<span class="p">.</span><span class="n">MinimumLevel</span><span class="p">.</span><span class="nf">Is</span><span class="p">(</span><span class="n">LogEventLevel</span><span class="p">.</span><span class="n">Verbose</span><span class="p">)</span>
<span class="p">.</span><span class="n">WriteTo</span><span class="p">.</span><span class="nf">RollingFile</span><span class="p">(</span><span class="s">"my-app-{Date}.log"</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Environment</span><span class="p">.</span><span class="n">UserInteractive</span><span class="p">)</span>
<span class="n">serilogConfiguration</span><span class="p">.</span><span class="n">WriteTo</span><span class="p">.</span><span class="nf">LiterateConsole</span><span class="p">();</span>
<span class="n">Log</span><span class="p">.</span><span class="n">Logger</span> <span class="p">=</span> <span class="n">serilogConfiguration</span><span class="p">.</span><span class="nf">CreateLogger</span><span class="p">();</span>
<span class="n">AppDomain</span><span class="p">.</span><span class="n">CurrentDomain</span><span class="p">.</span><span class="n">DomainUnload</span> <span class="p">+=</span> <span class="p">(</span><span class="n">o</span><span class="p">,</span> <span class="n">e</span><span class="p">)</span> <span class="p">=></span> <span class="n">Log</span><span class="p">.</span><span class="nf">CloseAndFlush</span><span class="p">();</span>
<span class="n">loggerFactory</span><span class="p">.</span><span class="nf">AddSerilog</span><span class="p">();</span>
<span class="n">provider</span><span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">SomeService</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">Run</span><span class="p">();</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>With the updated code, we get log messages pushed out to Visual Studio output window, a rolling file adjacent the exe, and if available, we will also get nicely coloured logs in a console window.</p>
<p><em>NB: The</em> <code class="highlighter-rouge">Environment.UserInteractive</code> <em>check may not be necessary for you, but we follow this practice to ensure we don’t attempt to output to a console when, for example, the application is running as a Windows Service.</em></p>
<h2 id="what-to-log">What to Log</h2>
<p>Now we know how to output logs, the next and most important question:</p>
<blockquote>
<p>What should I be logging?</p>
</blockquote>
<p>There is no simple answer to this, no rules that should be followed; the subject of logging will always be subjective. In my opinion though, there are 5 key components to useful logging…</p>
<h3 id="exceptions">Exceptions</h3>
<p>The first and most obvious thing to log is your exceptions. However, simply wrapping everything in a <code class="highlighter-rouge">try</code> / <code class="highlighter-rouge">catch</code> > <code class="highlighter-rouge">log</code> would be pretty wasteful and probably not as useful as you might think. I don’t want to dwell on exception management for too long, but I will mention a few things.</p>
<p>For starts, if you are in the habit of throwing exceptions for non <em>exceptional</em> circumstances, such as failed user input validation, you should stop doing this. If you were throwing an exception to produce log output, you now have an alternative. If you were throwing an exception to produce a different execution flow, this is called <em>exception logic</em> and is typically <a href="http://softwareengineering.stackexchange.com/a/107727/30301">considered a bad practice</a>.</p>
<p>Next up, don’t be afraid to allow exceptions to bubble up to a location that may have much more context regarding the current operation. Here’s a somewhat abstract example that hopefully demonstrates this point:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">async</span> <span class="n">Task</span> <span class="nf">DisableUser</span><span class="p">(</span><span class="n">Guid</span> <span class="n">userId</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">try</span>
<span class="p">{</span>
<span class="kt">var</span> <span class="n">user</span> <span class="p">=</span> <span class="k">await</span> <span class="n">_data</span><span class="p">.</span><span class="nf">GetUserAsync</span><span class="p">(</span><span class="n">userId</span><span class="p">);</span>
<span class="c1">// continue to disable user</span>
<span class="p">}</span>
<span class="k">catch</span> <span class="p">(</span><span class="n">DatabaseException</span> <span class="n">dex</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_logger</span><span class="p">.</span><span class="nf">LogError</span><span class="p">(</span><span class="m">0</span><span class="p">,</span> <span class="n">dex</span><span class="p">,</span> <span class="s">"Unable to disable user '{UserID}'"</span><span class="p">,</span> <span class="n">userId</span><span class="p">.</span><span class="nf">ToString</span><span class="p">(</span><span class="s">"N"</span><span class="p">));</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nf">DisableUserException</span><span class="p">(</span><span class="n">userId</span><span class="p">,</span> <span class="n">dex</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>In the above snippet, whilst my <code class="highlighter-rouge">_data</code> implementation <em>could</em> independently log the <code class="highlighter-rouge">DatabaseException</code>, I’ve decided the additional context available in the consumer would be better. I can therefore omit logging and even omit the <code class="highlighter-rouge">try</code>/<code class="highlighter-rouge">catch</code> in the data implementation (not shown here), and output the logs in the data consumer alongside information that we were attempting to disable a user when this exception occurred.</p>
<p><em>NB: For those wondering why I haven’t simply re-thrown the original exception after logging the output, this is a practice I try to follow. The</em> <code class="highlighter-rouge">DatabaseException</code> <em>is an <strong>implementation detail</strong> of the class shown. I don’t want consumers of this class to</em> <code class="highlighter-rouge">catch (DatabaseException)</code><em>, nor have to</em> <code class="highlighter-rouge">catch (Exception)</code><em>. The</em> <code class="highlighter-rouge">DisableUserException</code> <em>or something slightly more generic if preferred, allows the consumer to maintain cohesion and not depend upon this class’ dependencies directly.</em></p>
<h3 id="decisions">Decisions</h3>
<p>This is a lot trickier than logging exceptions, because you’ll need to decide what decisions are important to you. Everytime you write an <code class="highlighter-rouge">if</code> or <code class="highlighter-rouge">switch</code> statement, you probably branch the execution flow. Therefore, each time you find yourself writing an <code class="highlighter-rouge">if</code>/<code class="highlighter-rouge">switch</code>, consider whether it would be useful to output a trace message explaining such.</p>
<p>Whilst I believe this is good advice to follow, I can’t really provide examples; each case will be unique. I certainly don’t believe it necessary to output <em>every</em> decision made, but the odd <em>Trace</em> / <em>Debug</em> / <em>Verbose</em> log entry can be crucial in tracking down problems, especially if the issue arises in other environments that you’re not actively debugging.</p>
<h3 id="state">State</h3>
<p>Don’t forget to include useful state information in your log outputs. Yes, you can output the full stack trace for a particular exception, but how much easier would it be in replicating an exception if you also have the Customer ID, Order ID, etc. that was being processed at the time?</p>
<p>In addition to this, if you’ve followed my advice and added Serilog integration, you also gain access to <a href="https://github.com/serilog/serilog/wiki/Enrichment">Enrichers</a>. For example, I can add the following NuGet packages:</p>
<ul>
<li><a href="https://nuget.org/packages/serilog.enrichers.environment"><code class="highlighter-rouge">Serilog.Enrichers.Environment</code></a></li>
<li><a href="https://nuget.org/packages/serilog.enrichers.process"><code class="highlighter-rouge">Serilog.Enrichers.Process</code></a></li>
<li><a href="https://nuget.org/packages/serilog.enrichers.thread"><code class="highlighter-rouge">Serilog.Enrichers.Thread</code></a></li>
</ul>
<p>With these packages installed, I can update our application host to <em>Enrich</em> the the serilog configuration as shown here:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
</pre></td><td class="rouge-code"><pre><span class="kt">var</span> <span class="n">serilogConfiguration</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">LoggerConfiguration</span><span class="p">()</span>
<span class="p">.</span><span class="n">Enrich</span><span class="p">.</span><span class="nf">FromLogContext</span><span class="p">()</span>
<span class="p">.</span><span class="n">Enrich</span><span class="p">.</span><span class="nf">WithMachineName</span><span class="p">()</span>
<span class="p">.</span><span class="n">Enrich</span><span class="p">.</span><span class="nf">WithEnvironmentUserName</span><span class="p">()</span>
<span class="p">.</span><span class="n">Enrich</span><span class="p">.</span><span class="nf">WithProcessId</span><span class="p">()</span>
<span class="p">.</span><span class="n">Enrich</span><span class="p">.</span><span class="nf">WithThreadId</span><span class="p">()</span>
<span class="p">.</span><span class="n">MinimumLevel</span><span class="p">.</span><span class="nf">Is</span><span class="p">(</span><span class="n">LogEventLevel</span><span class="p">.</span><span class="n">Verbose</span><span class="p">)</span>
<span class="p">.</span><span class="n">WriteTo</span><span class="p">.</span><span class="nf">RollingFile</span><span class="p">(</span><span class="s">"my-app-{Date}.log"</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Whilst this enrichment won’t be particularly useful to the sinks so far mentioned, it can provide considerable value when combined with the recommendations in the <em>Advanced Sinks</em> section towards the end of this article.</p>
<h3 id="correlation">Correlation</h3>
<p>Nowadays, it’s common to be working with applications that are, in some fashion, able to process concurrent requests. When we’re writing code that handles a web request in an ASP.Net Website for example, we can often just think about that one request and not worry about how the framework might be managing multiple of these requests at the same time. However, this concurrency can have a profound effect on logging:</p>
<p>Request 1
<code class="highlighter-rouge">----> Logs Foo A -------------------------------------> Logs Bar F ---></code></p>
<p>Request 2
<code class="highlighter-rouge">-------> Logs Foo B -------------------------------> Logs Bar E ---></code></p>
<p>Request 3
<code class="highlighter-rouge">----------> Logs Foo C -------------------------> Logs Bar D ---></code></p>
<p>We simply cannot predict or rely upon even identical concurrent requests executing in the same order. In the above, although request 3 starts last, it finishes first. Why is this important? Consider your log output:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>Foo A
Foo B
Foo C
Bar D
Bar E
Bar F
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Without correlation, we have no idea that <code class="highlighter-rouge">Foo A</code> corresponds with <code class="highlighter-rouge">Bar F</code> and <code class="highlighter-rouge">Foo B</code> with <code class="highlighter-rouge">Bar E</code>, etc. This is usually very simple to resolve by adding a little bit more context:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
</pre></td><td class="rouge-code"><pre>Request 1 Foo A
Request 2 Foo B
Request 3 Foo C
Request 3 Bar D
Request 2 Bar E
Request 1 Bar F
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Now it’s much more clear which <code class="highlighter-rouge">Foo</code> and <code class="highlighter-rouge">Bar</code> messages belong together. It may seem like common sense now, but it’s all too easy to forget about correlation!</p>
<h3 id="performance">Performance</h3>
<p>The final subject for consideration are performance metrics. If you’re going to do this, I recommend doing so somewhat scarecly. For example, we log the time an overall request takes, which may involve any number of execution flows and thousands of lines of code, so provides only a very high level metric for the overall system performance. I don’t recommend using logging for lower level performance monitoring.</p>
<p>For more intricate performance monitoring, consider using <a href="http://benchmarkdotnet.org/">Benchmark.Net</a> to test smaller pieces of code in isolation, and/or outputting metrics using <a href="https://blogs.msdn.microsoft.com/vancem/2012/08/13/windows-high-speed-logging-etw-in-c-net-using-system-diagnostics-tracing-eventsource/">ETW</a>.</p>
<p>The point being, logging can be useful for highlighting an area that you should more thoroughly investigate, but the output should also be taken with a pinch of salt.</p>
<h2 id="what-level-to-log">What Level to Log</h2>
<p>There are only a couple pieces of advice I can give you on this subject.</p>
<h3 id="log-levels">Log Levels</h3>
<p>The first is to make sure you understand the order of the log levels in the frameworks you choose to use. For example, at one stage, the Microsoft Logging Abstractions used the following levels:</p>
<ul>
<li><code class="highlighter-rouge">Debug</code></li>
<li><code class="highlighter-rouge">Verbose</code></li>
<li><code class="highlighter-rouge">Information</code></li>
<li><code class="highlighter-rouge">Warning</code></li>
<li><code class="highlighter-rouge">Error</code></li>
<li><code class="highlighter-rouge">Critical</code></li>
</ul>
<p>However, <a href="https://github.com/aspnet/Announcements/issues/124">this was changed</a>, so the <strong>current</strong> Microsoft logging levels are as follows:</p>
<ul>
<li><code class="highlighter-rouge">Trace</code></li>
<li><code class="highlighter-rouge">Debug</code></li>
<li><code class="highlighter-rouge">Information</code></li>
<li><code class="highlighter-rouge">Warning</code></li>
<li><code class="highlighter-rouge">Error</code></li>
<li><code class="highlighter-rouge">Critical</code></li>
</ul>
<p>Whether you agree with the order of these levels or not, it is important to be aware of it. Similarly, the Serilog logging levels are:</p>
<ul>
<li><code class="highlighter-rouge">Verbose</code></li>
<li><code class="highlighter-rouge">Debug</code></li>
<li><code class="highlighter-rouge">Information</code></li>
<li><code class="highlighter-rouge">Warning</code></li>
<li><code class="highlighter-rouge">Error</code></li>
<li><code class="highlighter-rouge">Fatal</code></li>
</ul>
<p>As you can see, the updated Microsoft logging levels do map better to the Serilog levels, though they’re not identical. These log levels are typically used to <em>filter</em> your log outputs. For example, in local development and test environments, you may choose you want to output <em>all</em> of the logs, so you’ll set the Minimum log level in your application hosts to the lowest level available. However, in later stages of your deployment cycle you may choose to reduce the log spam of high throughput environments so that <em>Production</em> for example, may output only <code class="highlighter-rouge">Warning</code> and above.</p>
<p>Also make sure you’re aware what the <em>default</em> levels are of your logging frameworks, i.e. when no minimum level is configured or specified, what level will your framework use by default. For both Microsoft and Serilog, this tends to be <code class="highlighter-rouge">Information</code>.</p>
<h3 id="log-spam">Log Spam</h3>
<p>As alluded to in the previous section, <em>log spam</em> can be an issue. When writing new functionality I tend to add a lot of logging at <code class="highlighter-rouge">Trace</code> and <code class="highlighter-rouge">Debug</code> levels, knowing that running the code in a production environment would very quickly consume disk space and generally overload my sinks. Whilst testing, I’ll then try to decide which of my log entries have proven to be the most useful in either tracking down issues with the code, or in confirming that the code has worked as intended. Anything I deem to be less useful can either be changed to a lower log level, or removed completely if it provides no value.</p>
<p>In contrast, any log messages that prove to be crucial can have their log levels promoted. <strong>Make sure you test your log entries</strong> when you test your code! The last thing you want is to start outputting millions of log entries per second when your code hits production, bringing the application to a standstill or masking real problems!</p>
<h2 id="practices">Practices</h2>
<p>This next section will walk through some additional handy tips and tricks, some applicable to all situations whilst others are specific to the examples given. As per the foreword, you need to decide for yourself when these tips apply to your situation.</p>
<h3 id="interpolation">Interpolation</h3>
<p>One of the most common <em>mistakes</em> I’ve seen since the release of C#6 is people interpolating all of their strings. Don’t do this for logs! When using a proper logging framework, the <code class="highlighter-rouge">string.Format</code> style signatures will often store the arguments separately or display them differently. For example:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
</pre></td><td class="rouge-code"><pre><span class="kt">var</span> <span class="n">messageType</span> <span class="p">=</span> <span class="s">"Interpolated"</span><span class="p">;</span>
<span class="n">_logger</span><span class="p">.</span><span class="nf">LogInformation</span><span class="p">(</span><span class="s">$"This is an '</span><span class="p">{</span><span class="n">messageType</span><span class="p">}</span><span class="s">' log entry."</span><span class="p">);</span>
<span class="n">messageType</span> <span class="p">=</span> <span class="s">"Classic"</span><span class="p">;</span>
<span class="n">_logger</span><span class="p">.</span><span class="nf">LogInformation</span><span class="p">(</span><span class="s">"This is an '{messageType}' log entry."</span><span class="p">,</span> <span class="n">messageType</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The above will output to the Console (via <code class="highlighter-rouge">Serilog.Sinks.Literate</code>) as follows:</p>
<p><img src="https://puu.sh/tkXsf/3e945c79ad.png" alt="log output screenshot" /></p>
<p>Notice that the <em>interpolated</em> entry has no highlighting. That’s because it’s simply considered a part of the text, whereas the <em>classic</em> entry can be coloured, can be stored into queryable database columns, etc. This is because the parameters are available to the logger instead of being <em>“burned in”</em> at runtime, prior to reaching the logger. If you’ve ever had to regex through log files to find the lines you want, you’ll know exactly what I mean here.</p>
<h3 id="disposable-tracking">Disposable Tracking</h3>
<p>At last we finally reach the subject that drove this blog post. The application we’ve been working on was working perfectly in development and lower stages of our deployment cycle. However, when we put the application through load testing it completely fell apart.</p>
<p>Thanks to our application of the above logging practices, not to mention the advanced sink tips below, it was immediately obvious that we were not managing our disposable connections correctly. This particular issue is often very easy to find because most usages of <code class="highlighter-rouge">IDisposable</code> objects will be in a <code class="highlighter-rouge">using</code> statement. Given the right Roslyn analysers, you can simply check the Warnings page in VS and find the problem. However, the complexity of our application and the necessity to integrate with legacy code meant that in places we couldn’t simply wrap <code class="highlighter-rouge">using</code> statements around everything. For example, we use the <code class="highlighter-rouge">IServiceScopeFactory</code> to instantiate parts of the legacy code and rely on the Microsoft Dependency Injection container to track our transient disposables and tear them down for us when we <code class="highlighter-rouge">Dispose</code> the scope.</p>
<p>So, we needed a smarter way to track our connections, whilst the code that consumed said connections were essentially a black box to us given it could have been anywhere in millions of lines of heritage code.</p>
<p>Thanks to our aforementioned <em>service collection extensions</em> practice, the first step was simply to append a diagnostics connection factory to the <code class="highlighter-rouge">IServiceCollection</code>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
</pre></td><td class="rouge-code"><pre><span class="n">services</span><span class="p">.</span><span class="n">AddSingleton</span><span class="p"><</span><span class="n">DiagnosticsConnectionFactory</span><span class="p">>();</span>
<span class="n">services</span><span class="p">.</span><span class="nf">AddSingleton</span><span class="p">(</span><span class="n">provider</span> <span class="p">=></span> <span class="n">provider</span>
<span class="p">.</span><span class="n">GetRequiredService</span><span class="p"><</span><span class="n">DiagnosticsConnectionFactory</span><span class="p">>()</span>
<span class="p">.</span><span class="nf">CreateConnection</span><span class="p">());</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>The next step was to start logging when a connection was allocated, to what class the connection was allocated to, and of course, when that class released the connection. I’ve reduced the actual code used which also entailed pooling / throttling for sake of brevity, but the following should give you an idea of how it works:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
</pre></td><td class="rouge-code"><pre><span class="k">public</span> <span class="k">class</span> <span class="nc">DiagnosticsConnectionFactory</span>
<span class="p">{</span>
<span class="c1">// this is the connection provider that was originally</span>
<span class="c1">// used before we added this diagnostics provider.</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">OriginalConnectionProvider</span> <span class="n">_originalProvider</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">ILogger</span> <span class="n">_logger</span><span class="p">;</span>
<span class="k">public</span> <span class="nf">DiagnosticsConnectionFactory</span><span class="p">(</span>
<span class="n">OriginalConnectionProvider</span> <span class="n">originalProvider</span><span class="p">,</span>
<span class="n">ILogger</span><span class="p"><</span><span class="n">DiagnosticsConnectionFactory</span><span class="p">></span> <span class="n">logger</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">_originalProvider</span> <span class="p">=</span> <span class="n">originalProvider</span><span class="p">;</span>
<span class="n">_logger</span> <span class="p">=</span> <span class="n">logger</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="n">IDbConnection</span> <span class="nf">CreateConnection</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">// the int provided to the StackFrame constructor determines how many frames</span>
<span class="c1">// to skip up the stack. Zero would point to this method, each number greater</span>
<span class="c1">// to further up the stack. We found it most useful to move two frames up the</span>
<span class="c1">// stack because one frame would typically simply point us to the ctor of an</span>
<span class="c1">// entity framework context. What we actually wanted to see was the consumer</span>
<span class="c1">// of said context. Tweak this number or do something more intelligent as</span>
<span class="c1">// you deem fit.</span>
<span class="kt">var</span> <span class="n">stackFrame</span> <span class="p">=</span> <span class="k">new</span> <span class="n">System</span><span class="p">.</span><span class="n">Diagnostics</span><span class="p">.</span><span class="nf">StackFrame</span><span class="p">(</span><span class="m">2</span><span class="p">);</span>
<span class="kt">var</span> <span class="n">allocateeMethod</span> <span class="p">=</span> <span class="n">stackFrame</span><span class="p">.</span><span class="nf">GetMethod</span><span class="p">();</span>
<span class="kt">var</span> <span class="n">allocatee</span> <span class="p">=</span> <span class="s">$"</span><span class="p">{</span><span class="n">allocateeMethod</span><span class="p">.</span><span class="n">DeclaringType</span><span class="p">?.</span><span class="n">Name</span> <span class="p">??</span> <span class="s">"Unknown"</span><span class="p">}</span><span class="s">.</span><span class="p">{</span><span class="n">allocateeMethod</span><span class="p">.</span><span class="n">Name</span><span class="p">}</span><span class="s">"</span><span class="p">;</span>
<span class="kt">var</span> <span class="n">connection</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">DiagnosticsDbConnection</span><span class="p">(</span>
<span class="n">_originalProvider</span><span class="p">.</span><span class="n">CreateConnection</span><span class="p">,</span> <span class="n">DeallocateConnection</span><span class="p">,</span> <span class="n">allocatee</span><span class="p">);</span>
<span class="n">_logger</span><span class="p">.</span><span class="nf">LogDebug</span><span class="p">(</span><span class="s">"Database connection '{ConnectionID}' allocated to '{Allocatee}'. (Stack: '{StackFrame}')"</span><span class="p">,</span>
<span class="n">connection</span><span class="p">.</span><span class="n">Id</span><span class="p">,</span> <span class="n">connection</span><span class="p">.</span><span class="n">Allocatee</span><span class="p">,</span> <span class="n">stackFrame</span><span class="p">);</span>
<span class="k">return</span> <span class="n">connection</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">void</span> <span class="nf">DeallocateConnection</span><span class="p">(</span><span class="n">DiagnosticsDbConnection</span> <span class="n">connection</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">connection</span><span class="p">.</span><span class="nf">DisposeUnderlyingConnection</span><span class="p">();</span>
<span class="n">_logger</span><span class="p">.</span><span class="nf">LogDebug</span><span class="p">(</span><span class="s">"Database connection '{ConnectionID}' that was allocated to '{Allocatee}' has been released."</span><span class="p">,</span>
<span class="n">connection</span><span class="p">.</span><span class="n">Id</span><span class="p">,</span> <span class="n">connection</span><span class="p">.</span><span class="n">Allocatee</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">[</span><span class="nf">DebuggerDisplay</span><span class="p">(</span><span class="s">"Connection {"</span> <span class="p">+</span> <span class="k">nameof</span><span class="p">(</span><span class="n">Id</span><span class="p">)</span> <span class="p">+</span> <span class="s">"}"</span><span class="p">)]</span>
<span class="k">private</span> <span class="k">class</span> <span class="nc">DiagnosticsDbConnection</span> <span class="p">:</span> <span class="n">IDbConnection</span>
<span class="p">{</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">DbConnection</span><span class="p">></span> <span class="n">_connection</span><span class="p">;</span>
<span class="k">private</span> <span class="k">readonly</span> <span class="n">Action</span><span class="p"><</span><span class="n">DiagnosticsDbConnection</span><span class="p">></span> <span class="n">_deallocate</span><span class="p">;</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Id</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="kt">string</span> <span class="n">Allocatee</span> <span class="p">{</span> <span class="k">get</span><span class="p">;</span> <span class="p">}</span>
<span class="k">public</span> <span class="nf">DiagnosticsDbConnection</span><span class="p">(</span>
<span class="n">Func</span><span class="p"><</span><span class="n">DbConnection</span><span class="p">></span> <span class="n">connection</span><span class="p">,</span>
<span class="n">Action</span><span class="p"><</span><span class="n">DiagnosticsDbConnection</span><span class="p">></span> <span class="n">deallocate</span><span class="p">,</span>
<span class="kt">string</span> <span class="n">allocatee</span><span class="p">)</span>
<span class="p">{</span>
<span class="n">Id</span> <span class="p">=</span> <span class="n">Guid</span><span class="p">.</span><span class="nf">NewGuid</span><span class="p">().</span><span class="nf">ToString</span><span class="p">(</span><span class="s">"N"</span><span class="p">);</span>
<span class="n">_connection</span> <span class="p">=</span> <span class="k">new</span> <span class="n">Lazy</span><span class="p"><</span><span class="n">DbConnection</span><span class="p">>(</span><span class="n">connection</span><span class="p">);</span>
<span class="n">_deallocate</span> <span class="p">=</span> <span class="n">deallocate</span><span class="p">;</span>
<span class="n">Allocatee</span> <span class="p">=</span> <span class="n">allocatee</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// this is the crucial element, when the consumer attempts to Dispose</span>
<span class="c1">// the connection, it will actually just pass control back to our</span>
<span class="c1">// connection provider.</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Dispose</span><span class="p">()</span> <span class="p">=></span> <span class="nf">_deallocate</span><span class="p">(</span><span class="k">this</span><span class="p">);</span>
<span class="c1">// whether you're pooling or not, you still need the ability to</span>
<span class="c1">// dispose the actual connection.</span>
<span class="k">internal</span> <span class="k">void</span> <span class="nf">DisposeUnderlyingConnection</span><span class="p">()</span> <span class="p">=></span> <span class="n">_connection</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="nf">Dispose</span><span class="p">();</span>
<span class="c1">// the rest of this class contains what is hopefully an</span>
<span class="c1">// obvious pass-through of all IDbConnection members.</span>
<span class="c1">// for example:</span>
<span class="k">public</span> <span class="n">ConnectionState</span> <span class="n">State</span> <span class="p">=></span> <span class="n">_connection</span><span class="p">.</span><span class="n">Value</span><span class="p">.</span><span class="n">State</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>Whilst the above isn’t a practice that we will follow often, and certainly isn’t something I would want to deploy to Production, it very quickly highlighted our connection problems to us when we used this implementation under load.</p>
<h2 id="sinks-advanced">Sinks (Advanced)</h2>
<p>So, you’re armed with pretty much all the advice I can justify putting into this blog post at least. What else is there? So, so much more. If nothing else, before you leave, check out some of the very cool stuff you can do with some slightly more advanced sinks.</p>
<h3 id="seq">Seq</h3>
<p>Despite following <a href="https://twitter.com/nblumhardt">Nicholas Bulmhardt</a> on twitter for quite some time, I only recently became aware of <a href="https://getseq.net/"><em>Seq</em></a>. I’ve been waiting for our IT department to setup an <a href="https://www.elastic.co/webinars/introduction-elk-stack">ELK</a> instance for me, so decided to see if there were any simple alternatives I could get on with myself. Having spent less than a minute installing <em>Seq</em> to my local machine, I went ahead and added the <a href="https://nuget.org/packages/serilog.sinks.seq"><code class="highlighter-rouge">Serilog.Sinks.Seq</code></a> package and the single line of code required to enable it:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><table class="rouge-table"><tbody><tr><td class="rouge-gutter gl"><pre class="lineno">1
</pre></td><td class="rouge-code"><pre><span class="n">serilogConfiguration</span><span class="p">.</span><span class="n">WriteTo</span><span class="p">.</span><span class="nf">Seq</span><span class="p">(</span><span class="s">"http://localhost:5341"</span><span class="p">);</span>
</pre></td></tr></tbody></table></code></pre></div></div>
<p>I should point out, this blog post isn’t in anyway affiliated or sponsored by <em>Seq</em>, but I can’t praise it enough. It was so simple, and made our logs <em>so</em> accessible. With the above in place, you can jump into your <em>Seq</em> web portal and watch the logs live, filter events, write queries, and so much more. The <a href="http://docs.getseq.net/docs">documentation</a> is very thorough and easy to follow, the web interface easy to use, and the value added is immense. It does have a cost, but the guys at <em>Seq</em> were very helpful (actually extended our trial period for us whilst we waited for Finance to get the PO sorted).</p>
<p>My only gripe is, whilst the web portal is incredibly RESTful, generating URL’s for your queries and filters, the same does not appear to be true when you enable the live update. You’ll find yourself enabling the live feed constantly because everytime you expand a log entry or make a change, the page (correctly) disables the live update so the log entry you’re working with doesn’t scroll off the page. It would be great however to have a query value for this in the URL so that we can, for example, put the logs on an <em>information radiator</em> without having to configure an automatic refresh, or simply bookmark the feed with updates enabled.</p>
<p>One other feature that is definitely worth mentioning and enabling. Follow the guidance <a href="http://docs.getseq.net/docs/using-serilog#section-dynamic-level-control">shown here</a> to enable <em>Dynamic Level Control</em>. Once done, you can control the Minimum Log Levels we discussed earlier, at runtime, which again can prove immensely useful. Being able to temporarily change your Production environment from <code class="highlighter-rouge">Information</code> to <code class="highlighter-rouge">Debug</code> log levels while you’re trying to track down a difficult to reproduce issue is an awesome piece of functionality to have at your disposal, especially given how very simple it is to set up.</p>
<h2 id="summary">Summary</h2>
<p>As has been mentioned throughout this article, logging practices are mostly opinion and conjecture. I hope people find some of the information here useful, but don’t take it for gospel. Look around for other blogs and articles on logging (feel free to drop a comment below if you have a post you think I should link). For example, although it’s not mentioned here, our systems also make considerable use of <a href="https://azure.microsoft.com/en-gb/services/application-insights/">Application Insights</a>, <a href="https://analytics.google.com">Google Analytics</a>, and so forth.</p>
<p>Make sure you understand how your log frameworks and tooling work, and be sure to test your logging the same way you would test any piece of functionality you add to your code. Try to separate log output configuration from the code that actually produces log output, usually through use of abstractions. Finally, try not to <em>re-invent the wheel</em> with custom logging. There are simply too many high quality options out there to be wasting time and money <em>homebrewing</em>.</p>I tweeted a thank you to the guys over at Seq recently because Seq was instrumental in finding and eliminating a very difficult issue we came across. But technology choice alone isn’t enough, so I’ve decided to blog about some of the logging practices we tend to follow as a rule, and some specific practices that we used in a particularly complex application we’ve been developing over the last couple of months.Cleanspace Abstractions2016-12-03T00:00:00+00:002016-12-03T00:00:00+00:00http://blog.devbot.net/clean-space-abstractions