The easiest way to unload a .dll in powershell is to load (Import-Module) it up within a Job. You could also do it by creating an Appdomain, but a Job is the most PowerShell-y. Once the job completes, the .dll will get unloaded.

So this program isn't beautiful. What it does, is find the C# compiler on your machine (It will fail if its not there...). Compile a simple library. Load it. Call a function. Exit to job. Then, delete the .dll that it compiled. We can't delete a .dll while we have it loaded, so this is proof that the thing gets unloaded.

raw

# Create a job
$job = Start-Job -Name CsharpJob -ArgumentList $pwd -ScriptBlock {
    
    # Find the C sharp compiler
    $csc = "`"$((dir C:\ -File csc.exe -Recurse | Select -First 1).FullName)`" /target:library templib.cs"

    # Get the current working directory from the arguments
    cd $args[0]

    # Write a c# program to a temp file
@"
    namespace HelloWorldNamespace
    {
        public class HelloWorldLibrary
        {
            public static string GetHelloWorld()
            {
                return "Hello World";
            }
        }
    }
"@ > "templib.cs"

    # Build the program with csc
    iex "&$csc" | Out-Null

    # Import the dll we just built
    Import-Module "$pwd\templib.dll" | Out-Null

    # Call one of the methods in the libarary
    [HelloWorldNamespace.HelloWorldLibrary]::GetHelloWorld()

}

# Wait on the job to finish
Wait-Job -Job $job | Out-Null

# Write the output
Receive-Job -Job $job

# Clean up
rm templib.cs

# In a loop to to make sure this gets done
while(Test-Path templib.dll)
{
    rm templib.dll -ErrorAction Ignore | Out-Null
}

The reason for the loop at the end of the program, is that it takes a little bit of time for the OS to actually let you delete dll after it was unloaded. So this was kind of a hacky looking program, but it gets the job done!