Welcome to badflyer.com

This is a place where you can come to learn programming from a real professional programmer.

We really recommend you check out the latest endevour: https://logship.ai/

Await is just an operator

In .NET, and C# in particular, “await” is an operator just like ++ or &&. You can create all sorts of custom awaitable objects and build custom awaitable expressions.

Here is the simplest of examples!

Lets start by defining a task. A task needs to implement a single method:

  • GetAwaiter()
    • Returns an Awaiter. We will get to this next, its also got a schema to implement.
/// <summary>
/// A simple task.
/// </summary>
public class SimpleTask
{
    /// <summary>
    /// Gets the simple awaiter for the task.
    /// </summary>
    /// <returns>The awaiter.</returns>
    public SimpleAwaiter GetAwaiter()
    {
        return new SimpleAwaiter();
    }
}

Now for the Awaiter. The awaiter needs a few things defined:

  • GetResult()
    • Gets the result of the task.
    • Can be a void or non-void method.
  • IsCompleted
    • A property which returns a value indicating whether a task has completed. Ours is synchronous, so this always returns true.
  • ICriticalNotifyCompletion or INotifyCompletion
    • OnCompleted(Action continuation)
      • Called when returning from an async completion. If ICriticalNotifyCompletion is not implemented, then this will also be invoked on a synchronous completion.
    • UnsafeOnCompleted(Action continuation)
      • Only available if ICriticalNotifyCompletion is implemented, and will be invoked on a synchronous completion of the task. OnCompleted will not be called.
using System;
using System.Runtime.CompilerServices;

public class SimpleAwaiter : ICriticalNotifyCompletion, INotifyCompletion
{
    /// <summary>
    /// Our task never runs async, so its always completed.
    /// </summary>
    public bool IsCompleted => true;

    /// <summary>
    /// Our task doesnt return anything, so this can be a void method, 
    /// but lets return something so we have something to await.
    /// </summary>
    public string GetResult() => "Hello World";

    public void OnCompleted(Action continuation)
    {
        // OnCompleted is called on returning from an async completion.
        throw new NotImplementedException();
    }

    public void UnsafeOnCompleted(Action continuation)
    {
        // UnsafeOnCompleted is called after a sync completion.
        continuation();
    }
}

Heres an example of how this can all work!

class Program
{
    public static async Task Main(string[] args)
    {
        Console.WriteLine(await new SimpleTask());
    }
}

Give it a shot! The result will be “Hello World” in the console.

A good place to check for more information is the C# language reference: await-expressions

Deploying an NDIS HyperV Virtual Switch Extension

Here are some instructions on how to deploy a NDIS virtual switch extension to a Hyper-V Virtual Switch. This will save you some headaches during the driver deployment and validation process. Of course, before doing any of this, make sure you have a test host set up in Test Mode. “bcdedit /set testsigning on” Then reboot.

First comes first, after creating a basic NDIS lightweight filter driver project, make sure that your INF file is configured correctly. Here is a basic example, which will create a modifying filter driver which build for x64, and attaches only to virtual switches.

;-------------------------------------------------------------------------
; basicndis.INF -- NDIS LightWeight Filter Driver
;-------------------------------------------------------------------------

[version]
; Do not change these values
Signature       = "$Windows NT$"
Class           = NetService
ClassGUID       = {4D36E974-E325-11CE-BFC1-08002BE10318}
Provider        = %Badflyer%
DriverVer       = 
CatalogFile     = basicndis.cat


[Manufacturer]
%Badflyer%=MSFT,NTx86,NTamd64,NTarm,NTarm64

; BADFLYER_basicndis can be used with netcfg.exe to install/uninstall the driver.
[MSFT.NTx86]
%basicndis_Desc%=Install, BADFLYER_basicndis

[MSFT.NTamd64]
%basicndis_Desc%=Install, BADFLYER_basicndis

[MSFT.NTarm]
%basicndis_Desc%=Install, BADFLYER_basicndis

[MSFT.NTarm64]
%basicndis_Desc%=Install, BADFLYER_basicndis

;-------------------------------------------------------------------------
; Installation Section
;-------------------------------------------------------------------------
[Install]
AddReg=Inst_Ndi
; All LWFs must include the 0x40000 bit (NCF_LW_FILTER).
Characteristics=0x40000

; This must be a random, unique value.
; FILTER_UNIQUE_NAME in filter.h must match this GUID identically.
; Both should have {curly braces}.
NetCfgInstanceId="{3ca735b3-e816-470b-905e-9d5097241c74}"

Copyfiles = basicndis.copyfiles.sys

[SourceDisksNames]
1=%basicndis_Desc%,"",,

[SourceDisksFiles]
basicndis.sys=1

[DestinationDirs]
DefaultDestDir=12
basicndis.copyfiles.sys=12

[basicndis.copyfiles.sys]
basicndis.sys,,,2


;-------------------------------------------------------------------------
; Ndi installation support
;-------------------------------------------------------------------------
[Inst_Ndi]
HKR, Ndi,Service,,"basicndis"
HKR, Ndi,CoServices,0x00010000,"basicndis"
HKR, Ndi,HelpText,,%basicndis_HelpText%

HKR, Ndi,FilterClass,, "ms_switch_filter"
; TODO: Specify whether you have a Modifying or Monitoring filter.
; For a Monitoring filter, use this:
;     HKR, Ndi,FilterType,0x00010001, 1 ; Monitoring filter
; For a Modifying filter, use this:
;     HKR, Ndi,FilterType,0x00010001, 2 ; Modifying filter
HKR, Ndi,FilterType,0x00010001,2
; Do not change these values. These are required for a vswitch filter driver.
HKR, Ndi\Interfaces,UpperRange,,"noupper"
HKR, Ndi\Interfaces,LowerRange,,"nolower"
; In order to work with the virtual switch, you must include "vmnetextension". 
; Can also include "ethernet" to work on regular network stacks.
HKR, Ndi\Interfaces, FilterMediaTypes,,"vmnetextension"
; TODO: Specify whether you have a Mandatory or Optional filter
; For a Mandatory filter, use this:
;     HKR, Ndi,FilterRunType,0x00010001, 1 ; Mandatory filter
; For an Optional filter, use this:
;     HKR, Ndi,FilterRunType,0x00010001, 2 ; Optional filter
; Optional filters will allow the network stack on continue working if the filter is not.
HKR, Ndi,FilterRunType,0x00010001, 2 ; Optional filter

;-------------------------------------------------------------------------
; Service installation support
;-------------------------------------------------------------------------
[Install.Services]
; 0x800 Means to start the service automatically after installation. Remove it if you do not want that.
AddService=basicndis,0x800,basicndis_Service_Inst

[basicndis_Service_Inst]
DisplayName     = %basicndis_Desc%
ServiceType     = 1 ;SERVICE_KERNEL_DRIVER
; Typically you will want your filter driver to start with SERVICE_SYSTEM_START.
; If it is an Optional filter, you may also use 3;SERVICE_DEMAND_START.
StartType       = 1 ;SERVICE_SYSTEM_START
ErrorControl    = 1 ;SERVICE_ERROR_NORMAL
ServiceBinary   = %12%\basicndis.sys
LoadOrderGroup  = NDIS
Description     = %basicndis_Desc%
AddReg          = NdisImPlatformBindingOptions.reg

[Install.Remove.Services]
; The SPSVCINST_STOPSERVICE flag instructs SCM to stop the NT service
; before uninstalling the driver.
DelService=basicndis,0x200 ; SPSVCINST_STOPSERVICE

[NdisImPlatformBindingOptions.reg]
; By default, when an LBFO team or Bridge is created, all filters will be
; unbound from the underlying members and bound to the TNic(s). This keyword
; allows a component to opt out of the default behavior
; To prevent binding this filter to the TNic(s):
;   HKR, Parameters, NdisImPlatformBindingOptions,0x00010001,1 ; Do not bind to TNic
; To prevent unbinding this filter from underlying members:
;   HKR, Parameters, NdisImPlatformBindingOptions,0x00010001,2 ; Do not unbind from Members
; To prevent both binding to TNic and unbinding from members:
;   HKR, Parameters, NdisImPlatformBindingOptions,0x00010001,3 ; Do not bind to TNic or unbind from Members
HKR, Parameters, NdisImPlatformBindingOptions,0x00010001,0 ; Subscribe to default behavior



[Strings]
; TODO: Customize these strings.
Badflyer = "badflyer" ;TODO: Replace with your manufacturer name
basicndis_Desc = "basicndis NDIS LightWeight Filter"
basicndis_HelpText = "basicndis NDIS LightWeight Filter"

The comments here are mostly from the NDIS lightweight filter template which comes with the Windows Driver Kit. Now you can install the driver onto a target computer. Assuming the target computer is a 64 bit machine.

; The important sections to note from the .info file:

; This specifies the x64 install, and we will need 'BADFLYER_basicndis' to install with netcfg
[MSFT.NTamd64]
%basicndis_Desc%=Install, BADFLYER_basicndis

; This specifies that this is a filtering extension
HKR, Ndi,FilterClass,, "ms_switch_filter"

; This specifies that we will bind to a virtual switch as an extension
HKR, Ndi\Interfaces, FilterMediaTypes,,"vmnetextension"

; 0x800 Automatically starts the driver after installation.
AddService=basicndis,0x800,basicndis_Service_Inst
  1. Compile the project as x64
  2. Copy the output to the target computer. (The target computer should bet in testmode “bcdedit /set testsigning on”). The output directory should contain atleast 3 files.
    • basicndis.cat
    • basicndis.inf
    • basicndis.sys
  3. Use netcfg to install the driver. (instructions below)
  4. Use powershell to enable the extension on the virtual switch (instructions below)

So, now that the files are copied over. You can use netcfg.exe to install the driver service. This will come by default on windows.

#
# You can lookup the documentation for netcfg online, but here is basically what needs to happen:
# netcfg /l <path to inf file> /c S /i <driver installation name from inf>
#
# The driver installation name can be found/set in the .inf file in the platform configuration section.
# EX:  ; BADFLYER_basicndis can be used with netcfg.exe to install/uninstall the driver.
#        [MSFT.NTx86]
#        %basicndis_Desc%=Install, BADFLYER_basicndis
#
# Here is an example
netcfg /l C:\Users\Administrator\Desktop\basicndis\basicndis.inf /c S /i BADFLYER_basicndis

If all goes well, you will get a nice happy message about success. If it does not, you will get an error code. Logs for netcfg can be found under “C:\Windows\INF\setupapi.dev.log” aka “%SYSTEMROOT%\INF\setupapi.dev.log” and “%SYSTEMROOT%\INF\setupapi.app.log”.

Hopefully as is well, can you have gotten this far, you can enable the extension on the Hyper-V virtual switch. In this example, I have a VM Switch named “InternalSwitch”.

PS C:\Users\Administrator> Get-VMSwitchExtension -VMSwitchName internalswitch | where { $_.Vendor -match 'badflyer' }


Id                  : 3CA735B3-E816-470B-905E-9D5097241C74
Name                : basicndis NDIS LightWeight Filter
Vendor              : badflyer
Version             : 23.54.47.252
ExtensionType       : Filter
ParentExtensionId   :
ParentExtensionName :
SwitchId            : 655c9bd4-0d5b-4322-8b39-1b9a58e0ce94
SwitchName          : InternalSwitch
Enabled             : False
Running             : True
CimSession          : CimSession: .
ComputerName        : WIN-7Q9KPM774O8
IsDeleted           : False

If you query for it, the driver is running, but is not enabled on the switch. But that’s an easy fix.

Get-VMSwitchExtension -VMSwitchName internalswitch | where { $_.Vendor -match 'badflyer' } | Enable-VMSwitchExtension
# or
Get-VMSwitchExtension -VMSwitchName internalswitch | where { $_.Vendor -match 'badflyer' } | Disable-VMSwitchExtension

That’s all there is to it. After that, your NDIS filter driver will begin to receive traffic from the virtual switch, and will be part of the virtual switch’s driver stack.

# To uninstall the driver, simply use netcfg#
#
# netcfg /u <driver installation name from inf>
#
netcfg /u BADFLYER_basicndis

To start and stop the driver server, you can use:

# net start <name of service>
# net stop <name of service>
# Stop-Service <name of service>
# Start-Service <name of service>

# EX: (Note, in this example this is not the same as the name given to netcfg)
#    You can make them the same if you configure your inf that way, but the service
#    name is not necessarily the same as the name of the section used for installation.
net start basicndis

MSBuild Cheat Sheet

Welcome to the msbuild cheat sheet. This is a list of some of the things I always need to search for when I am creating an MSBuild project .props/.targets file. The cheat sheet is in the form of a .proj file, which can be executed by MSBuild.

If you’re ever having troubles with MSBuild, try “msbuild /verbosity:diagnostic”, which will make MSBuild print out its entire knowledge of the world.

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <!--
        How to define a variable.
            Just stick a new node in a property group.
    -->
    <PropertyGroup>
        <!-- This node in a property group will define a variable -->
        <TestVariable>Test Variable Value</TestVariable>
        
        <!-- Adding a condition, which checks if the variable is already defined,
             will allow you to override the variable in projects. 
             If the variable is not defined, it will evaulate to an empty string
             within the condition and be set with the value defined here.-->
        <TestVariableWithOverride Condition="'$(TestVariableWithOverride)' == ''">Test override.</TestVariableWithOverride>
    </PropertyGroup>

    <Target Name="main" DependsOnTargets="PrintMyVariables;PrintMSBuildVariables;Conditions;PrintPropertyFunctions;PrintOutBatching">
        <!-- Msbuild will process the first target in the file by default.
            By creating this target, and making it depend on the two following targets,
            we can ensure that they will all be executed
            -->
    </Target>

    <!--
        How to print a message.
    -->
    <Target Name="PrintMyVariables">
        <!-- Prints a basic message -->
        <Message Text="Here is my message.."></Message>
        
        <!-- Message importance. How messages are shown can change with the logger being used.
        These examples use the default console logger.-->
        
        <!-- Shows with atleast (Normal) verbosity -->
        <Message Text="Low importance" Importance="Low" />
        
        <!-- Shows with atleast (Normal) verbosity -->
        <Message Text="Normal importance (This is the default)" Importance="Normal" />
        
        <!-- Shows with atleast (Minimal) verbosity -->
        <Message Text="High importance" Importance="High" />
        
        <!-- Interpolates the value of the test variable into the string. -->
        <Message Text="My test variable: $(TestVariable)" />
    </Target>
    
    <!--
        Standard msbuild variables.
    -->
    <Target Name="PrintMSBuildVariables">
        <Message Text="MSBuildAssemblyVersion               -> '$(MSBuildAssemblyVersion)'             " />
        
        <!-- The absolute path of the folder where the MSBuild binaries that are currently being used are located -->
        <Message Text="MSBuildBinPath                       -> '$(MSBuildBinPath)'                     " />
        
        <!-- The path of the MSBuild subfolder under the \Program Files or \Program Files (x86) folder.
        This path always points to the 32-bit \Program Files folder on a 32-bit machine and \Program Files (x86)
        on a 64-bit machine. These two properties are the same-->
        <Message Text="MSBuildExtensionsPath                -> '$(MSBuildExtensionsPath)'              " />
        <Message Text="MSBuildExtensionsPath32              -> '$(MSBuildExtensionsPath32)'            " />
        
        <!-- The path of the MSBuild subfolder under the \Program Files folder.
        For a 64-bit machine, this path always points to the \Program Files folder.
        For a 32-bit machine, this path is blank. -->
        <Message Text="MSBuildExtensionsPath64              -> '$(MSBuildExtensionsPath64)'            " />
        
        <!-- Paths to the .net framework folders, if they are installed -->
        <Message Text="MSBuildFrameworkToolsPath            -> '$(MSBuildFrameworkToolsPath)'          " />
        <Message Text="MSBuildFrameworkToolsPath32          -> '$(MSBuildFrameworkToolsPath32)'        " />
        <Message Text="MSBuildFrameworkToolsPath64          -> '$(MSBuildFrameworkToolsPath64)'        " />
        <Message Text="MSBuildFrameworkToolsRoot            -> '$(MSBuildFrameworkToolsRoot)'          " />
        
        
        <Message Text="MSBuildLoadMicrosoftTargetsReadOnly  -> '$(MSBuildLoadMicrosoftTargetsReadOnly)'" />
        
        <!-- The maximum number of concurrent processes that are used when building.
        This is the value that you specified for /maxcpucount on the command line.
        If you specified /maxcpucount without specifying a value, then MSBuildNodeCount
        specifies the number of processors in the computer. -->
        <Message Text="MSBuildNodeCount                     -> '$(MSBuildNodeCount)'                   " />

        <Message Text="MSBuildProgramFiles32                -> '$(MSBuildProgramFiles32)'              " />
        <Message Text="MSBuildProjectDirectory              -> '$(MSBuildProjectDirectory)'            " />
        <Message Text="MSBuildProjectDirectoryNoRoot        -> '$(MSBuildProjectDirectoryNoRoot)'      " />
        <Message Text="MSBuildProjectExtension              -> '$(MSBuildProjectExtension)'            " />
        <Message Text="MSBuildProjectFile                   -> '$(MSBuildProjectFile)'                 " />
        <Message Text="MSBuildProjectFullPath               -> '$(MSBuildProjectFullPath)'             " />
        <Message Text="MSBuildProjectName                   -> '$(MSBuildProjectName)'                 " />
        <Message Text="MSBuildRuntimeType                   -> '$(MSBuildRuntimeType)'                 " />
        <Message Text="MSBuildRuntimeVersion                -> '$(MSBuildRuntimeVersion)'              " />
        <Message Text="MSBuildSDKsPath                      -> '$(MSBuildSDKsPath)'                    " />
        <Message Text="MSBuildStartupDirectory              -> '$(MSBuildStartupDirectory)'            " />
        
        <!-- Gets the current file. -->
        <Message Text="MSBuildThisFile                      -> '$(MSBuildThisFile)'                    " />
       
        <!-- Gets the current file directory. -->
        <Message Text="MSBuildThisFileDirectory             -> '$(MSBuildThisFileDirectory)'           " />

        <Message Text="MSBuildThisFileDirectoryNoRoot       -> '$(MSBuildThisFileDirectoryNoRoot)'     " />
        <Message Text="MSBuildThisFileExtension             -> '$(MSBuildThisFileExtension)'           " />
        <Message Text="MSBuildThisFileFullPath              -> '$(MSBuildThisFileFullPath)'            " />
        <Message Text="MSBuildThisFileName                  -> '$(MSBuildThisFileName)'                " />
        <Message Text="MSBuildToolsPath                     -> '$(MSBuildToolsPath)'                   " />
        <Message Text="MSBuildToolsPath32                   -> '$(MSBuildToolsPath32)'                 " />
        <Message Text="MSBuildToolsPath64                   -> '$(MSBuildToolsPath64)'                 " />
        <Message Text="MSBuildToolsRoot                     -> '$(MSBuildToolsRoot)'                   " />
        <Message Text="MSBuildToolsVersion                  -> '$(MSBuildToolsVersion)'                " />
        <Message Text="MSBuildUserExtensionsPath            -> '$(MSBuildUserExtensionsPath)'          " />
        <Message Text="MSBuildVersion                       -> '$(MSBuildVersion)'                     " />
    </Target>

    <!-- Condition tests. -->
    <Target Name="Conditions">
        <!-- String equality -->
        <Message Condition="yellow == 'yellow'"     Text="Quotes not required for one word.    -> yellow == 'yellow" />
        <Message Condition="YELLOW == yellow"       Text="Case is insensitive.                 -> YELLOW == yellow" />
        <Message Condition="red != blue"            Text="Not equals works too.                -> red != blue" />
        
        <!-- String unary operators -->
        <Message Condition="Exists('$(MSBuildProjectFullPath)')" Text="Checks if the file or folder exists. -> Exists('$(MSBuildProjectFullPath)')" />
        <Message Condition="HasTrailingSlash('test\')"           Text="Checks for a trailing slash /.       -> HasTrailingSlash('test\')" />

        <!-- Logical operators -->
        <message Condition="true AND true"          Text="AND operator                         -> true AND true" />
        <message Condition="true OR false"          Text="OR operator                          -> true OR false" />
        <message Condition="!false"                 Text="NOT operator                         -> !false" />
        <message Condition="(true AND false) OR true"   Text="Grouping works                       -> (true AND false) OR true" />
    </Target>
    
    <!-- Property Functions (You can nest them)-->
    <Target Name="PrintPropertyFunctions">
        <!-- There are lots of these. Most of them are just totally regular .net classes. Thanks MSDN-->
        <Message Text="The syntax to get a property is [Class]::Property. For example $([System.Int32]::MaxValue)" />
        
        <Message Text="Any method or property from          -> System.Byte                                  " />
        <Message Text="Any method or property from          -> System.Char                                  " />
        <Message Text="Any method or property from          -> System.Convert                               " />
        <Message Text="Any method or property from          -> System.DateTime                              " />
        <Message Text="Any method or property from          -> System.Decimal                               " />
        <Message Text="Any method or property from          -> System.Double                                " />
        <Message Text="Any method or property from          -> System.Enum                                  " />
        <Message Text="Any method or property from          -> System.Guid                                  " />
        <Message Text="Any method or property from          -> System.Int16                                 " />
        <Message Text="Any method or property from          -> System.Int32                                 " />
        <Message Text="Any method or property from          -> System.Int64                                 " />
        <Message Text="Any method or property from          -> System.IO.Path                               " />
        <Message Text="Any method or property from          -> System.Math                                  " />
        <Message Text="Any method or property from          -> System.UInt16                                " />
        <Message Text="Any method or property from          -> System.UInt32                                " />
        <Message Text="Any method or property from          -> System.UInt64                                " />
        <Message Text="Any method or property from          -> System.SByte                                 " />
        <Message Text="Any method or property from          -> System.Single                                " />
        <Message Text="Any method or property from          -> System.String                                " />
        <Message Text="Any method or property from          -> System.StringComparer                        " />
        <Message Text="Any method or property from          -> System.TimeSpan                              " />
        <Message Text="Any method or property from          -> System.Text.RegularExpressions.Regex         " />
        <Message Text="Any method or property from          -> Microsoft.Build.Utilities.ToolLocationHelper " />

        <Message Text="These methods work too               -> System.Environment::CommandLine                " />
        <Message Text="These methods work too               -> System.Environment::ExpandEnvironmentVariables " />
        <Message Text="These methods work too               -> System.Environment::GetEnvironmentVariable     " />
        <Message Text="These methods work too               -> System.Environment::GetEnvironmentVariables    " />
        <Message Text="These methods work too               -> System.Environment::GetFolderPath              " />
        <Message Text="These methods work too               -> System.Environment::GetLogicalDrives           " />
        <Message Text="These methods work too               -> System.IO.Directory::GetDirectories            " />
        <Message Text="These methods work too               -> System.IO.Directory::GetFiles                  " />
        <Message Text="These methods work too               -> System.IO.Directory::GetLastAccessTime         " />
        <Message Text="These methods work too               -> System.IO.Directory::GetLastWriteTime          " />
        <Message Text="These methods work too               -> System.IO.Directory::GetParent                 " />
        <Message Text="These methods work too               -> System.IO.File::Exists                         " />
        <Message Text="These methods work too               -> System.IO.File::GetCreationTime                " />
        <Message Text="These methods work too               -> System.IO.File::GetAttributes                  " />
        <Message Text="These methods work too               -> System.IO.File::GetLastAccessTime              " />
        <Message Text="These methods work too               -> System.IO.File::GetLastWriteTime               " />
        <Message Text="These methods work too               -> System.IO.File::ReadAllText                    " />
    
        <!-- Combining Paths
        This can be a little annoying, because you're never really sure if a variable includes the final backslash.
        To Get around this issue, you can use the regular Path.combine -->
        <Message Text="System.IO.Path::Combine              -> $([System.IO.Path]::Combine('C:\This\Is\How\', 'You\Combine\Paths', 'to\a\file.txt'))" />
        
        <!-- Special MSBuild operators -->
        <Message Text="Add two values(double/int/long)      -> [MSBuild]::Add(1.5, 2.5) = $([MSBuild]::Add(1.5, 2.5))" />
        <Message Text="Subtract two values(double/int/long) -> [MSBuild]::Subtract(7, 9) = $([MSBuild]::Subtract(7, 9))" />
        <Message Text="Multiply two values(double/int/long) -> [MSBuild]::Multiply(5, 4) = $([MSBuild]::Multiply(5, 4))" />
        <Message Text="Divide two values(double/int/long)   -> [MSBuild]::Divide(8, 2) = $([MSBuild]::Divide(8, 2))" />
        <Message Text="Modulo two values(double/int/long)   -> [MSBuild]::Modulo(42, 5) = $([MSBuild]::Modulo(42, 5))" />
        
        <!-- Haven't exactly figured out where to use these escape functions yet.. -->
        <Message Text="Escape a string                      -> [MSBuild]::Escape(' a% b$ c@ d; e? f* ') = $([MSBuild]::Escape(' a% b$ c@ d; e? f* '))" />
        <Message Text="Unescape a string                    -> [MSBuild]::Unescape('%25 %24 %40 %27 %3B %3F %2A') = $([MSBuild]::Unescape('%25 %24 %40 %27 %3B %3F %2A'))" />
        
        <!-- Bitwise operations are supported -->
        <Message Text="Bitwise Or                           -> [MSBuild]::BitwiseOr(1, 2) = $([MSBuild]::BitwiseOr(1, 2))" />
        <Message Text="Bitwise And                          -> [MSBuild]::BitwiseAnd(3, 1) = $([MSBuild]::BitwiseAnd(3, 1))" />
        <Message Text="Bitwise Xor                          -> [MSBuild]::BitwiseXor(1, 1) = $([MSBuild]::BitwiseXor(1, 1))" />
        <Message Text="Bitwise Not                          -> [MSBuild]::BitwiseNot(0) = $([MSBuild]::BitwiseNot(0))" />
        
        <!-- Nested example -->
        <Message Text="Nested example                       -> [MSBuild]::Subtract([MSBuild]::Add(10, 5), 7) = $([MSBuild]::Subtract([MSBuild]::Add(10, 5), 7)" />
    </Target>

    <!-- Test directory-->
    <PropertyGroup> 
        <!-- Test directory-->
        <TestDirectory>$([System.IO.Path]::Combine('$(TMP)', 'bftestfiles\'))</TestDirectory>
    </PropertyGroup>

    <!-- Create a directory -->
    <Target Name="CreateTestDirectory">
        <MakeDir Directories="$(TestDirectory)" />
    </Target>
    
    <!-- Delete a directory, along with all files inside -->
    <Target Name="DeleteTestDirectory" AfterTargets="PrintOutBatching">
        <RemoveDir Directories="$(TestDirectory)" />
    </Target>

    <!-- Write to a text file -->
    <Target Name="CreateTestFiles" DependsOnTargets="CreateTestDirectory">
        <WriteLinesToFile File="$([System.IO.Path]::Combine('$(TestDirectory)', '1-test.txt'))" Overwrite="True" Lines="Test 1" />
        <WriteLinesToFile File="$([System.IO.Path]::Combine('$(TestDirectory)', '2-test.txt'))" Lines="Test 2" />
    </Target>

    <!-- Perform an action for each item in a list -->
    <Target Name="PrintOutBatching" DependsOnTargets="CreateTestFiles">
    
        <!-- An item group which finds the .txt files we created in the test folder.
        A dynamic item group like this one, is evalued when the task is run.
        An item group declared directly under the project is evaluated when the project is loaded.
        Any files which are created during the build would only show up in a dynamic item group. -->
        <ItemGroup>
            <WindowsDll Include="$(TMP)\**\*test.txt"></WindowsDll>
        </ItemGroup>

        <!-- Each one of these is executed for each unique value found. Because of the '%' sign. 
        Notice that identical values like RootDir are only printed once.-->
        <Message Text="Full path                            ->  FullPath = %(WindowsDll.FullPath)" />
        <Message Text="Root dir                             ->  RootDir = %(WindowsDll.RootDir)" />
        <Message Text="The file name                        ->  Filename = %(WindowsDll.Filename)" />
        <Message Text="The extension                        ->  Extension = %(WindowsDll.Extension)" />
        <Message Text="The relative directory (include path)->  RelativeDir = %(WindowsDll.RelativeDir)" />
        <Message Text="The full file directory              ->  Directory = %(WindowsDll.Directory)" />
        <Message Text="Recursive directory (only if \**\)   ->  RecursiveDir = %(WindowsDll.RecursiveDir)" />
        <Message Text="Identity (Path from include)         ->  Identity = %(WindowsDll.Identity)" />
        <Message Text="The modified time                    ->  ModifiedTime = %(WindowsDll.ModifiedTime)" />
        <Message Text="The created time                     ->  CreatedTime = %(WindowsDll.CreatedTime)" />
        <Message Text="The accessed time                    ->  AccessedTime = %(WindowsDll.AccessedTime)" />
    </Target>
</Project>

T-SQL SHA256

Sometimes things just make more sense in SQL… Sometimes they don’t. Here is a working implementation of the SHA2 algorithm written in T-SQL.

-- If you are here looking for code to actually use in production, here you go, no need to keep reading:

SELECT HASHBYTES('sha2_256', 'The quick brown fox jumps over the lazy dog')

To get things started, SQL doesn’t support binary shifts by default. So here are some helper functions to do bit shifting. SHA256 requires a right shift, and a right rotation. (Right rotation in turn requires a left shift). Shift are implemented by doing power of 2 multiplication/division. Every time you multiply by two, you do a single left shift. Every time you divide by two, you do a single right shift.

--
-- SHA256 implementation
--
CREATE PROCEDURE SHA256
(
    @input VARBINARY(MAX)
)
AS BEGIN
    SET NOCOUNT ON
    SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

    DECLARE @kvalues TABLE
    (
        id INT,
        val BIGINT
    )

    -- Initialize all of the K values
    INSERT INTO @kvalues
    VALUES 
     (0,  0x428a2f98),(1,  0x71374491),(2,  0xb5c0fbcf),(3,  0xe9b5dba5)
    ,(4,  0x3956c25b),(5,  0x59f111f1),(6,  0x923f82a4),(7,  0xab1c5ed5)
    ,(8,  0xd807aa98),(9,  0x12835b01),(10, 0x243185be),(11,0x550c7dc3)
    ,(12, 0x72be5d74),(13, 0x80deb1fe),(14, 0x9bdc06a7),(15, 0xc19bf174)
    ,(16, 0xe49b69c1),(17, 0xefbe4786),(18, 0x0fc19dc6),(19, 0x240ca1cc)
    ,(20, 0x2de92c6f),(21, 0x4a7484aa),(22, 0x5cb0a9dc),(23, 0x76f988da)
    ,(24, 0x983e5152),(25, 0xa831c66d),(26, 0xb00327c8),(27, 0xbf597fc7)
    ,(28, 0xc6e00bf3),(29, 0xd5a79147),(30, 0x06ca6351),(31, 0x14292967)
    ,(32, 0x27b70a85),(33, 0x2e1b2138),(34, 0x4d2c6dfc),(35, 0x53380d13)
    ,(36, 0x650a7354),(37, 0x766a0abb),(38, 0x81c2c92e),(39, 0x92722c85)
    ,(40, 0xa2bfe8a1),(41, 0xa81a664b),(42, 0xc24b8b70),(43, 0xc76c51a3)
    ,(44, 0xd192e819),(45, 0xd6990624),(46, 0xf40e3585),(47, 0x106aa070)
    ,(48, 0x19a4c116),(49, 0x1e376c08),(50, 0x2748774c),(51, 0x34b0bcb5)
    ,(52, 0x391c0cb3),(53, 0x4ed8aa4a),(54, 0x5b9cca4f),(55, 0x682e6ff3)
    ,(56, 0x748f82ee),(57, 0x78a5636f),(58, 0x84c87814),(59, 0x8cc70208)
    ,(60, 0x90befffa),(61, 0xa4506ceb),(62, 0xbef9a3f7),(63, 0xc67178f2)

    DECLARE @hvalues TABLE
    (
        a BIGINT
       ,b BIGINT
       ,c BIGINT
       ,d BIGINT
       ,e BIGINT
       ,f BIGINT
       ,g BIGINT
       ,h BIGINT
    )

    -- Initialize the initial H values
    INSERT INTO @hvalues
    VALUES
    (0x6a09e667
    ,0xbb67ae85
    ,0x3c6ef372
    ,0xa54ff53a
    ,0x510e527f
    ,0x9b05688c
    ,0x1f83d9ab
    ,0x5be0cd19)

    DECLARE @loop INT = 0

    -- Length of the input binary in bytes
    DECLARE @inputLength BIGINT = LEN(@input)

    -- Total length of the binary, including the 1 bit to append to the data, and the 8 byte length of the data
    DECLARE @totalLength BIGINT = @inputLength + 1 + 8

    -- Length that needs to padded with zeros
    DECLARE @padLength BIGINT = 64 - (@totalLength % 64)

    -- If the result is evenly divisible, than we don't need to pad
    SET @padLength = CASE WHEN @padLength = 64 THEN 0 ELSE @padLength END

    -- Declare the array of zeros we need to append, and pad with the right number of zeros.
    DECLARE @zeros VARBINARY(55) = CONVERT(VARBINARY(MAX), REPLICATE(CHAR(0), @padLength))

    -- Create the full length padded binary
    DECLARE @binary VARBINARY(MAX) = @input + 0x80 + @zeros + CONVERT(VARBINARY(8), (@inputLength * 8))

    DECLARE @messageSchedules TABLE
    (
        chunk INT
       ,id    INT
       ,val   BIGINT
    )

    -- Split the data into 4 byte words
    -- Copy 64 bytes (16 words) into each 64 word chunk
    -- The other 48 bits are all zero, they need to be calculated later.
    --
    -- Result:
    -- chunk | id | value
    ----------------------
    --   0   |  0 | 4 BYTES FROM index (0)
    --   0   |  1 | 4 BYTES FROM index (4)
    --          .
    --          .
    --   0   | 15 | 4 BYTES FROM index (60)
    --   0   | 16 | 0x00000000
    --          .
    --          .
    --   0   | 63 | 0x00000000
    --   1   |  0 | 4 BYTES FROM index (64)
    --          .
    --          .
    --   1   | 16 | 0x00000000
    --   1   | 17 | 0x00000000
   ;WITH splitter AS
    (
        SELECT
          0 AS id
         ,1 AS pos
         ,SUBSTRING(@binary, 1, 4) AS val
        UNION ALL
        SELECT
          s.id + 1 AS id
         ,CASE WHEN (((s.id + 1) % 64) < 16) THEN s.pos + 4 ELSE s.pos END
         ,CASE WHEN (((s.id + 1) % 64) < 16) THEN SUBSTRING(@binary, s.pos + 4, 4)
               ELSE 0x00000000
               END AS val
        FROM splitter s
        WHERE s.id + 1 < len(@binary)
    ) 
    INSERT INTO @messageSchedules
    SELECT
        s.id / 64 AS chunk
       ,s.id % 64 AS id
       ,CONVERT(BIGINT, s.val)
    FROM splitter s OPTION (MAXRECURSION 0)

    DECLARE @rows BIGINT = @@ROWCOUNT

    -- Sadly this bit I couldn't figure out how to make more SQL-y yet. 
    -- For every value, you need access to previously updated values in the table,
    -- Which doesn't work, since you cant see all of the values which you have updated.
    -- 
    -- We need to calculate the working values (everything we set to zero above, so anything
    --   with an id larger than 15)
    -- 
    -- for every id 16 <= i < 64
    --    @messageSchedules[i] = 0xFFFFFFFF 
    --              &amp; (@messageSchedules[i-16] 
    --                  + (RIGHTROTATE(@messageSchedules[i-15], 7) ^ RIGHTROTATE(@messageSchedules[i-15], 18) ^ RIGHTSHIFT(@messageSchedules[i-15], 3))
    --                  + @messageSchedules[i-7] 
    --                  + (RIGHTROTATE(@messageSchedules[i-2], 17) ^ RIGHTROTATE(@messageSchedules[i-2], 19) ^ RIGHTSHIFT(@messageSchedules[i-2], 10)))
    SET @loop = 16
    WHILE @loop < @rows * 64
    BEGIN

        UPDATE ws
            SET ws.val = 0xFFFFFFFF 
                    &amp; (ws16.val 
                    + (dbo.ROTR32(ws15.val, 7) 
                        ^ dbo.ROTR32(ws15.val,18) 
                        ^ dbo.SHR32(ws15.val, 3)) 
                    + ws7.val 
                    + (dbo.ROTR32(ws2.val, 17)
                        ^ dbo.ROTR32(ws2.val, 19)
                        ^ dbo.SHR32(ws2.val, 10)))
        FROM @messageSchedules ws
        JOIN @messageSchedules ws16
          ON ws.chunk = ws16.chunk AND ws.id - 16 = ws16.id
        JOIN @messageSchedules ws2
          ON ws.chunk = ws2.chunk AND ws.id - 2 = ws2.id
        JOIN @messageSchedules ws7
          ON ws.chunk = ws7.chunk AND ws.id - 7 = ws7.id
        JOIN @messageSchedules ws15
          ON ws.chunk = ws15.chunk AND ws.id - 15 = ws15.id
        WHERE ws.id >= 16 AND ws.id = @loop

        SET @loop = @loop + 1
    END

    -- This is the meat of it. 
    -- We start with row -1, which is made up of the h values. (ai,bi...hi) store the "initial" h value for the 64 byte set.
    -- a,b,c,d,e,g,h store the working values. They are all of the temp variables
    -- ai,bi,ci...hi store the base value for each 64 byte set of temp variables. At every new set of 64 bytes, we start with ai,bi,ci...
    --  At the end of every 64 bytes (a.id + 1) % 64 = 0, we store the new set of base values so we can add them back in at the end. (a.id + 1) % 64 = 63
   ;WITH abcdefgh AS
    (
        SELECT -1 AS id, a, b, c, d, e, f, g, h, a AS ai, b AS bi, c AS ci, d AS di, e AS ei, f AS fi, g AS gi, h AS hi
        FROM @hvalues

        UNION ALL

        SELECT
            a.id + 1 AS id,
            ((dbo.ROTR32(a.e, 6) ^ dbo.ROTR32(a.e, 11) ^ dbo.ROTR32(a.e, 25))      -- Sigma 1
                + ((a.e &amp; a.f) ^ ((~a.e) &amp; a.g))                                   -- Ch (choice function)
                + a.h                                                              
                + (SELECT val FROM @kvalues WHERE id = ((a.id + 1) % 64))          
                + bc.val                                                           
                + ((a.a &amp; a.b) ^ (a.a &amp; a.c) ^ (a.b &amp; a.c))                        -- Maj (Majority Function)
                + (dbo.ROTR32(a.a, 2) ^ dbo.ROTR32(a.a, 13) ^ dbo.ROTR32(a.a, 22)) -- Sigma 0
                + CASE WHEN (a.id + 1) % 64 = 63 THEN a.ai ELSE 0 END)
                &amp; 0xFFFFFFFF AS a,
            (a.a + CASE WHEN (a.id + 1) % 64 = 63 THEN a.bi ELSE 0 END) &amp; 0xFFFFFFFF AS b,
            (a.b + CASE WHEN (a.id + 1) % 64 = 63 THEN a.ci ELSE 0 END) &amp; 0xFFFFFFFF AS c,
            (a.c + CASE WHEN (a.id + 1) % 64 = 63 THEN a.di ELSE 0 END) &amp; 0xFFFFFFFF AS d,
            (a.d + (dbo.ROTR32(a.e, 6) ^ dbo.ROTR32(a.e, 11) ^ dbo.ROTR32(a.e, 25)) -- Sigma 1
                + ((a.e &amp; a.f) ^ ((~a.e) &amp; a.g))                                    -- Ch (choice function)
                + a.h 
                + (SELECT val FROM @kvalues WHERE id = ((a.id + 1) % 64))
                + bc.val
                + CASE WHEN (a.id + 1) % 64 = 63 THEN a.ei ELSE 0 END)
                &amp; 0xFFFFFFFF AS e,
            (a.e + CASE WHEN (a.id + 1) % 64 = 63 THEN a.fi ELSE 0 END) &amp; 0xFFFFFFFF AS f,
            (a.f + CASE WHEN (a.id + 1) % 64 = 63 THEN a.gi ELSE 0 END) &amp; 0xFFFFFFFF AS g,
            (a.g + CASE WHEN (a.id + 1) % 64 = 63 THEN a.hi ELSE 0 END) &amp; 0xFFFFFFFF AS h
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.a ELSE a.ai END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.b ELSE a.bi END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.c ELSE a.ci END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.d ELSE a.di END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.e ELSE a.ei END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.f ELSE a.fi END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.g ELSE a.gi END
            ,CASE WHEN (a.id + 1) % 64 = 0 THEN a.h ELSE a.hi END
        FROM abcdefgh a
        JOIN @messageSchedules bc ON bc.id = ((a.id + 1) % 64) AND bc.chunk = ((a.id + 1) / 64)
        CROSS JOIN @hvalues hv
    )
    SELECT TOP 1
            CONVERT(VARBINARY(4), a)
           + CONVERT(VARBINARY(4), b)
           + CONVERT(VARBINARY(4), c)
           + CONVERT(VARBINARY(4), d)
           + CONVERT(VARBINARY(4), e)
           + CONVERT(VARBINARY(4), f)
           + CONVERT(VARBINARY(4), g)
           + CONVERT(VARBINARY(4), h)
    FROM abcdefgh 
    ORDER BY id DESC OPTION (MAXRECURSION 0)

  END
GO

DECLARE @input VARBINARY(MAX) = CONVERT(VARBINARY(MAX), 'The quick brown fox jumps over the lazy dog')
EXECUTE SHA256 @input = @input -- Should be: 'd7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592'

SET @input = CONVERT(VARBINARY(MAX), '')
EXECUTE SHA256 @input = @input -- Should be: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

And there you have it! SHA2 implemented in SQL. Most of the algorithm is implemented in the giant CTE (abcdefgh) at the bottom of the file. If you do it completely procedural-y (while loops), you can just follow the regular Wikipedia (https://en.wikipedia.org/wiki/SHA-2) example and get a pretty clear solution.

--
-- Procedural SHA256 implementation
--
CREATE PROCEDURE SHA256
(
    @input VARBINARY(MAX)
)
AS BEGIN
    SET NOCOUNT ON

    DECLARE @kvalues TABLE
    (
        id INT,
        val BIGINT
    )

    INSERT INTO @kvalues
    VALUES 
     (0,  0x428a2f98),(1,  0x71374491),(2,  0xb5c0fbcf),(3,  0xe9b5dba5)
    ,(4,  0x3956c25b),(5,  0x59f111f1),(6,  0x923f82a4),(7,  0xab1c5ed5)
    ,(8,  0xd807aa98),(9,  0x12835b01),(10, 0x243185be),(11,0x550c7dc3)
    ,(12, 0x72be5d74),(13, 0x80deb1fe),(14, 0x9bdc06a7),(15, 0xc19bf174)
    ,(16, 0xe49b69c1),(17, 0xefbe4786),(18, 0x0fc19dc6),(19, 0x240ca1cc)
    ,(20, 0x2de92c6f),(21, 0x4a7484aa),(22, 0x5cb0a9dc),(23, 0x76f988da)
    ,(24, 0x983e5152),(25, 0xa831c66d),(26, 0xb00327c8),(27, 0xbf597fc7)
    ,(28, 0xc6e00bf3),(29, 0xd5a79147),(30, 0x06ca6351),(31, 0x14292967)
    ,(32, 0x27b70a85),(33, 0x2e1b2138),(34, 0x4d2c6dfc),(35, 0x53380d13)
    ,(36, 0x650a7354),(37, 0x766a0abb),(38, 0x81c2c92e),(39, 0x92722c85)
    ,(40, 0xa2bfe8a1),(41, 0xa81a664b),(42, 0xc24b8b70),(43, 0xc76c51a3)
    ,(44, 0xd192e819),(45, 0xd6990624),(46, 0xf40e3585),(47, 0x106aa070)
    ,(48, 0x19a4c116),(49, 0x1e376c08),(50, 0x2748774c),(51, 0x34b0bcb5)
    ,(52, 0x391c0cb3),(53, 0x4ed8aa4a),(54, 0x5b9cca4f),(55, 0x682e6ff3)
    ,(56, 0x748f82ee),(57, 0x78a5636f),(58, 0x84c87814),(59, 0x8cc70208)
    ,(60, 0x90befffa),(61, 0xa4506ceb),(62, 0xbef9a3f7),(63, 0xc67178f2)

    DECLARE @hvalues TABLE
    (
        id INT,
        val BIGINT
    )

    INSERT INTO @hvalues
    VALUES
     (0, 0x6a09e667)
    ,(1, 0xbb67ae85)
    ,(2, 0x3c6ef372)
    ,(3, 0xa54ff53a)
    ,(4, 0x510e527f)
    ,(5, 0x9b05688c)
    ,(6, 0x1f83d9ab)
    ,(7, 0x5be0cd19)

    -- Initialize temp variables
    DECLARE @a BIGINT
    DECLARE @b BIGINT
    DECLARE @c BIGINT
    DECLARE @d BIGINT
    DECLARE @e BIGINT
    DECLARE @f BIGINT
    DECLARE @g BIGINT
    DECLARE @h BIGINT
    DECLARE @s1 BIGINT
    DECLARE @s0 BIGINT
    DECLARE @ch BIGINT
    DECLARE @maj BIGINT
    DECLARE @temp1 BIGINT
    DECLARE @temp2 BIGINT
    DECLARE @loop INT

    -- Length of the input binary in bytes
    DECLARE @inputLength BIGINT = LEN(@input)

    -- Total length of the binary, including the 1 bit to append to the data, and the 8 byte length of the data
    DECLARE @totalLength BIGINT = @inputLength + 1 + 8

    -- Length that needs to padded with zeros
    DECLARE @padLength BIGINT = 64 - (@totalLength % 64)

    -- If the result is evenly divisible, than we don't need to pad
    SET @padLength = CASE WHEN @padLength = 64 THEN 0 ELSE @padLength END

    -- Declare the array of zeros we need to append, and pad with the right number of zeros.
    DECLARE @zeros VARBINARY(55) = CONVERT(VARBINARY(MAX), REPLICATE(CHAR(0), @padLength))

    -- Create the full length padded binary
    DECLARE @binary VARBINARY(MAX) = @input + 0x80 + @zeros + CONVERT(VARBINARY(8), (@inputLength * 8))

    DECLARE @allDataLoop INT = 0
    DECLARE @binLength INT = DATALENGTH(@binary)

    DECLARE @messageSchedule TABLE
    (
        id INT,
        val BIGINT
    )

    WHILE (@allDataLoop <> @binLength)
    BEGIN
        DELETE FROM @messageSchedule

        -- Get all of the working bytes. 
       ;WITH cte AS
        (
            SELECT
                0 AS id
               ,SUBSTRING(@binary, @allDataLoop + 0 + 1, 4) AS val

            UNION ALL
        
            SELECT
                c.id + 1 AS id
               ,SUBSTRING(@binary, @allDataLoop + ((c.id + 1) * 4) + 1, 4) AS val
            FROM cte AS c
            WHERE c.id < 15
        )
        INSERT INTO @messageSchedule
        SELECT
          id,
          val
        FROM cte

        -- Calculate the values with indicies larger than 16 in the set of working bytes
        SET @loop = 16
        WHILE (@loop < 64)
        BEGIN
            INSERT INTO @messageSchedule
            SELECT
                @loop AS id
               ,0xFFFFFFFF &amp; (ws16.val + (dbo.ROTR32(ws15.val, 7) ^ dbo.ROTR32(ws15.val,18) ^ dbo.SHR32(ws15.val, 3)) + ws7.val + (dbo.ROTR32(ws2.val, 17) ^ dbo.ROTR32(ws2.val, 19) ^ dbo.SHR32(ws2.val, 10))) AS val
            FROM @messageSchedule ws15
            JOIN @messageSchedule ws2
              ON ws2.id = @loop-2
            JOIN @messageSchedule ws16
              ON ws16.id = @loop-16
            JOIN @messageSchedule ws7
              ON ws7.id = @loop-7
            WHERE ws15.id = @loop-15

            SET @loop = @loop + 1
        END

        -- Initialize the temp variables
        SELECT @a = val FROM @hvalues WHERE id = 0
        SELECT @b = val FROM @hvalues WHERE id = 1
        SELECT @c = val FROM @hvalues WHERE id = 2
        SELECT @d = val FROM @hvalues WHERE id = 3
        SELECT @e = val FROM @hvalues WHERE id = 4
        SELECT @f = val FROM @hvalues WHERE id = 5
        SELECT @g = val FROM @hvalues WHERE id = 6
        SELECT @h = val FROM @hvalues WHERE id = 7

        SET @loop = 0
        WHILE (@loop < 64)
        BEGIN
            SET @s1 = (dbo.ROTR32(@e, 6) ^ dbo.ROTR32(@e, 11) ^ dbo.ROTR32(@e, 25)) &amp; 0xFFFFFFFF -- Sigma 1
            SET @ch = ((@e &amp; @f) ^ ((~@e) &amp; @g)) &amp; 0xFFFFFFFF                                    -- Ch (choice function)
            SET @temp1 = (@h + @s1 + @ch + (SELECT val FROM @kvalues WHERE id = @loop) + (SELECT val FROM @messageSchedule WHERE id = @loop)) &amp; 0xFFFFFFFF

            SET @s0 = (dbo.ROTR32(@a, 2) ^ dbo.ROTR32(@a, 13) ^ dbo.ROTR32(@a, 22)) &amp; 0xFFFFFFFF -- Sigma 0
            SET @maj = (@a &amp; @b) ^ (@a &amp; @c) ^ (@b &amp; @c)                                         -- Maj (Majority Function)
            SET @temp2 = (@s0 + @maj) &amp; 0xFFFFFFFF

            SET @h = @g
            SET @g = @f
            SET @f = @e
            SET @e = (@d + @temp1) &amp; 0xFFFFFFFF
            SET @d = @c
            SET @c = @b
            SET @b = @a
            SET @a = (@temp1 + @temp2) &amp; 0xFFFFFFFF

            SET @loop = @loop + 1
        END

        -- Update the base values
        UPDATE @hvalues SET val = (val + @a) &amp; 0xFFFFFFFF WHERE id = 0 
        UPDATE @hvalues SET val = (val + @b) &amp; 0xFFFFFFFF WHERE id = 1
        UPDATE @hvalues SET val = (val + @c) &amp; 0xFFFFFFFF WHERE id = 2
        UPDATE @hvalues SET val = (val + @d) &amp; 0xFFFFFFFF WHERE id = 3
        UPDATE @hvalues SET val = (val + @e) &amp; 0xFFFFFFFF WHERE id = 4
        UPDATE @hvalues SET val = (val + @f) &amp; 0xFFFFFFFF WHERE id = 5
        UPDATE @hvalues SET val = (val + @g) &amp; 0xFFFFFFFF WHERE id = 6
        UPDATE @hvalues SET val = (val + @h) &amp; 0xFFFFFFFF WHERE id = 7

        SET @allDataLoop = @allDataLoop + 64
    END

    DECLARE @result VARBINARY(MAX) = 0x
    SELECT @result += CONVERT(VARBINARY(4), val) FROM @hvalues
    SELECT @result
END
GO

DECLARE @input VARBINARY(MAX) = CONVERT(VARBINARY(MAX), 'The quick brown fox jumps over the lazy dog')
EXECUTE SHA256 @input = @input -- Should be: 'd7a8fbb307d7809469ca9abcb0082e4f8d5651e46d3cdb762d02d0bf37c9e592'

SET @input = CONVERT(VARBINARY(MAX), '')
EXECUTE SHA256 @input = @input -- Should be: 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'

Clearly not as cool, but the procedural solution is pretty much textbook and much easier to follow.

VMDX to VHD conversion

You can easily convert a VMWare VMDX file to a Microsoft VHD file using the “Microsoft Virtual Machine Converter“. The Command ‘ConvertTo-MvmcVirtualHardDisk’ transforms the VMWare disk into a Hyper-V Compatible disk.

Once the Microsoft Virtual Machine Converter is installed, you can load the commandlets into a powershell session:

Import-Module 'C:\Program Files\Microsoft Virtual Machine Converter\MvmcCmdlet.psd1'

Once you have that, a command like below will do the trick:

ConvertTo-MvmcVirtualHardDisk -SourceLiteralPath 'D:\VM\CNC Windows 7 Professional\Windows 7 Pro
fessional-cl1.vmdk' -DestinationLiteralPath 'D:\VM\CNC Windows 7 Professional Hyper-v\Windows 7 Professional-cl1.vhd' -V
hdType DynamicHardDisk -VhdFormat Vhd

Errors

Now, for the true purpose of this article. Sometimes, ‘ConvertTo-MvmcVirtualHardDisk’ will throw an error, because it can’t understand something in the descriptor of the .VMDX. An error like below…

ConvertTo-MvmcVirtualHardDisk -SourceLiteralPath 'D:\VM\CNC Windows 7 Professional\Windows 7 Pro
fessional-cl1.vmdk' -DestinationLiteralPath 'D:\VM\CNC Windows 7 Professional Hyper-v\Windows 7 Professional-cl1.vhd' -V
hdType DynamicHardDisk -VhdFormat Vhd
ConvertTo-MvmcVirtualHardDisk : The entry 1 is not a supported disk database entry for the descriptor.
At line:1 char:1
+ ConvertTo-MvmcVirtualHardDisk -SourceLiteralPath 'D:\VM\CNC Windows 7 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (Microsoft.Accel...nversionService:DriveConversionService) [ConvertTo-MvmcVi
   rtualHardDisk], VmdkDescriptorParseException
    + FullyQualifiedErrorId : DiskConversion,Microsoft.Accelerators.Mvmc.Cmdlet.Commands.ConvertToMvmcVirtualHardDiskCommand

ConvertTo-MvmcVirtualHardDisk : One or more errors occurred.
At line:1 char:1
+ ConvertTo-MvmcVirtualHardDisk -SourceLiteralPath 'D:\VM\CNC Windows 7 ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : WriteError: (Microsoft.Accel...nversionService:DriveConversionService) [ConvertTo-MvmcVi
   rtualHardDisk], AggregateException
    + FullyQualifiedErrorId : DiskConversion,Microsoft.Accelerators.Mvmc.Cmdlet.Commands.ConvertToMvmcVirtualHardDiskCommand

Finding good instructions online for how to solve this were tough, and most included downloading another program. This script takes the VMDK file, and reads the 1024 byte file descriptor at offset 512, then writes it to a text file. It will open that file in notepad++, but you can use any editor you like. After editing the text file, save, and use the second half of the script to write it back into the VMDK. If the VMDK is very important to you, please make a copy, to make sure you can still access your data if something were to go wrong.

# Open VM-ware disk, read 1024 bytes at position 512
$vmdkFileName = 'D:\VM\CNC Windows 7 Professional\Windows 7 Professional-cl1.vmdk'
$vmdkFileStream = [System.IO.File]::Open($vmdkFileName, [System.IO.FileMode]::Open, [System.IO.FileAccess]::ReadWrite)
$vmdkFileStream.Position = 512

$bytes = [byte[]]::new(1024);
$vmdkFileStream.Read($bytes, 0, 1024)

# Write to a temp file
$tempPath = [System.IO.Path]::Combine([System.IO.Path]::GetTempPath(), [System.IO.Path]::GetTempFileName())
$tempfile = [System.IO.File]::OpenWrite($tempPath)
$tempfile.Write($bytes, 0, 1024)
$tempfile.Dispose()

# Open the editor. Wait for exit doesn't always seem to work for npp...
# Use whichever edit you like, it needs to show text, and also helpful if it can show whitespace/control characters
$editor = Start-Process 'C:\Program Files\Notepad++\notepad++.exe' -ArgumentList $tempPath -PassThru -Wait
$editor.WaitForExit()

# TODO, change what is causing the problem in the opened file.

Now, an editor will open. You will see the file descriptor. The error message will describe which line in the file is causing the issue. Commenting (with a ‘#’) that line out of the file did the trick for me. Save, and run this to write the contents back into the VMDK.

For example, my error was “ConvertTo-MvmcVirtualHardDisk : The entry 1 is not a supported disk database entry for the descriptor.” So I commented out the line: db.toolsInstallType = “1”.

# Read back the temp file
$tempfile = [System.IO.File]::OpenRead($tempPath)
$tempfile.Read($bytes, 0, 1024);
$tempfile.Dispose()

# Write back to the vmdk
$vmdkFileStream.Position = 512
$vmdkFileStream.Write($bytes, 0, 1024)

# Cleanup
$vmdkFileStream.Dispose();
del $tempPath

Then try the conversion again!. If the entry to change was ambiguous, and didnt work, you can run the same steps again to try different changes.

Implementing The Kusto Query Language

This is the first bit on a series of implementing a big data search engine, and the beginning of a series on implementing the backend services of logship. The goal is to implement a search / analytics system that anybody could walk off the street and use. The first few posts will start with getting the query language right, since that'll be the entry point to most users. For comparison some of the most popular Big Data search systems out there use SQL as their query language. Think Presto or Spark. SQL is fantastic, but generally a very difficult language to use for non experts; espessially as the queries get complicated.

In order to lower barriers to entry for logship, I didn't think SQL was the way to go. While it is one of the most popular languages in the world, the syntax makes it clunky for data analytics. (Another interesting side note, when SQL is the entry point, users seem to treat the query engine as if they are actually querying a SQL database, a distinction I'd like to avoid.)

Kusto to the Rescue! In the example below, I have a table of the posts scrapped from the /r/politics subreddit. Following are a few examples of why Kusto was the choice made.

reddit_politics
| take 100

Equivalent SQL

SELECT *
FROM reddit_politics
LIMIT 100

Kusto is a completely sequential language. Every operation is applied on top of the previous one (flowing downwards in the query). So looking at, and understanding a query is extremely simple.

reddit_politics
| where ups > 10000
| where title contains "biden"
| where title contains "arizona"
| take 100

Pretty simple right? reddit_politics -> posts with more than 10000 up votes -> take pots where the title contains the word 'biden' -> further filter down to posts where the title contains the word 'arizona' -> limit to 100 results. The equivalent SQL would look like this:

SELECT *
FROM reddit_politics
WHERE ups > 10000
  AND title contains 'biden'
  AND title contains 'arizona'
LIMIT 100

So this is the intro, stay tuned for more!

Simple kusto query

Also investing on running the entire query stack out of a command line, for testability and quick offline analysis. Sporting all the same features, optimizations, and performances.

Simple kusto CLI

You can find some more information on the logship blog, about how to use kusto operators and expressions. For example: Kusto Where

vfzip

We had a deployment issue with Logship. The size of the deployment directory had ballooned with the addition of each new .net core service. At around 5GB, that size was so prohibatively large that build artifact movement between the build servers and artifact storage + build artifact download to production nodes was taking many minutes.

Logship build output

The Problem

.Net Core self-contained executables are big. The publish process packages up a lot of the framework, so that it is available for the executable at runtime. There are a couple of options here:

  • Assembly Trimming (SLOW)
  • Install the .Net Core Runtime (ANNOYING)

So what's the issue. As the number of microservices increases, the number of self-contained executables increases. We end up with the same libraries published into the execution directory of each microservice.

Zip file

First attempt at shrinking the size was to zip up the entire folder for deployment. This worked pretty well at the beginning, but zip performance was bad, and final file size was large. The Zip algorithem also doesn't deduplicate at the file level, just the blocks of content. In the end, the 5GB directory compresses down to about: 1.8GB.

Solution

The issue here is the number of duplicate, but idential, files. vfzip is designed to fix the duplicate file compression problem specifically, acheiving much higher compression ratios.

Process - Compression

  1. First Phase: We hash every file in the target directory, so we can make sure we only store each file one time.
  2. Second Phase: We compress unique files via the Deflate algorithm into a container, indexed by the SHA1 file hash, encoding the directory layout at the end.

Process - Decompression

  1. Inflate Phase: All unique files are extracted from the Deflated file onto disk.
  2. Rebuild Phase: The directory tree is reconstructed from the extracted files. There are two options here, either the extracted files can be copied into their target folders with their new names, or the directory tree can be recreated via symlinks, saving space.

vfzip

vfzip is able to reduce the size of our build artifacts from 5GB to 104MB. Since we only compress each file once, the vfzip container creation process is significantly faster than a zip for directories which contain duplicate files.

Source code

https://github.com/petersulucz/vfzip

mdbook via nix!

Nix as a development environment is truely fantastic. Perfectly reproduceable, and gone are are the "run this script" setup steps.

This webpage is generated by mdbook, but even further so the environment is a nix flake, so the setup is really trivial.

Configuration

This is the flake.nix which makes it all happen:

{
  description = "Badflyer.com";
  nixConfig.bash-prompt = "\[nix-badflyer-shell\]$ ";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let 
        pkgs = nixpkgs.legacyPackages.${system};
        inputs = [
          pkgs.mdbook
          pkgs.git
        ];

        builder = pkgs.stdenv.mkDerivation {
          name = "badflyer";
          src = ./.;
          buildInputs = inputs;
          buildPhase = ''
            mdbook build
          '';
          installPhase = ''
            mkdir $out
            cp -R book/* $out
          '';
        };

      in {
        devShell = pkgs.mkShell {
          buildInputs = inputs;
          };

        packages.default = builder;
        defaultPackage = builder;
        checks.default = builder;
      });
}

The build workflow is as simple as:

git add -all && nix build

The development workflow is as simple as:

nix develop
mdbook serve

Deployment

This is a github pages site. Unfortunatly the free github runners don't have any nix support. So how does it get deployed? The Magic is in git subtree split, which allow you to commit a folder into the root of a different branch.

# This is a basic workflow to help you get started with Actions

name: CI

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the "main" branch
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v3

      # Runs a single command using the runners shell
      - name: Install mdbook
        run: |
          mkdir bin
          # Download mdbook
          curl -sSL https://github.com/rust-lang/mdBook/releases/download/v0.4.18/mdbook-v0.4.18-x86_64-unknown-linux-gnu.tar.gz | tar -xz --directory=bin
          # build the site
          bin/mdbook build
          # Specific to github, make sure the cname file ends up in the deployment. For the custom domain name.
          cp CNAME book/CNAME

      # Runs a set of commands using the runners shell
      - name: Deployment
        run: |
          git config --global user.email "gh-deploy@github.com"
          git config --global user.name "Continuous Integration"
          # Force add the book folder (this is the output of mdbook build)
          git add book -f
          # Commit (this will go onto master, but it's fine, we're throwing it away.)
          git commit -m "Updating"
          # Commit the 'book' folde onto the branch gh-pages
          git subtree split -P book -b gh-pages
          # Force push gh-pages
          git push --force origin gh-pages

About

This is a github pages site. No data is collected.