Mike DeLuca 5/11/17 Mike DeLuca 5/11/17

Azure BLOB Storage and PowerShell: The Hard Way

Shared Key Authentication Scheme

In a previous post I covered my general love/hate affair with PowerShell; particularly with respect to the Microsoft Cloud. For the majority of you than can not be bothered to read, I expressed a longstanding grudge against the Azure Cmdlets, rooted in the Switch-AzureMode fiasco. As an aside, those of you enjoying the Azure Stack technical previews may notice as similar problem arising with 'AzureRM Profile', but I digress. More importantly, there was a general theme of understanding the abstractions placed in front of you as an IT professional. By now, most of you should be familiar with the OAuth Bearer tokens used throughout the Microsoft cloud. They are nearly ubiquitous, with the exception of a few services, most importantly storage. The storage service is authenticated with a Shared Key Authentication or a Shared Access Signature. I will be focusing on the former.

Anatomy of the Signature

The Authentication header of HTTP requests backing the Azure Storage Services take the following form:

Authorization: SharedKey <Storage Account Name>:<AccessSignature>

The access signature is an HMAC 256 encoded string (Signature) which is constructed mostly of the components of the backing HTTP request. The gritty details are (somewhat) clearly detailed at MSDN, but for example the string to be encoded for getting the list of blobs in a container, looks something like this.

GET
x-ms-date:Mon, 08 May 2017 23:28:20 GMT x-ms-version:2016-05-31 /nosaashere/certificates comp:list restype:container

Let's examine the properties of a request for creating a BLOB Snapshot.

GET https://nosaashere.blob.core.windows.net/nosaashere/managedvhds/Provisioned.vhdx?comp=snapshot

Canonical Resource comp:snapshot Canonical Resource Query

PUT VERB x-ms-date:Mon, 08 May 2017 23:28:21 GMT Canonical Date Header x-ms-version:2016-05-31 Canonical Header /nosaashere/managedvhds/Provisioned.vhdx

A more advanced request (like this example for appending data to a Page BLOB) will show how additional headers come into scope as we include an MD5 Hash to verify the content, a content-length, and other required API headers.

PUT
4096000
32qczJv1wUlqnJPQRdBUzw==
x-ms-blob-type:PageBlob
x-ms-date:Mon, 08 May 2017 23:28:39 GMT
x-ms-page-write:Update x-ms-range:bytes=12288000-16383999
x-ms-version:2016-05-31 /nosaashere/managedvhds/Provisioned.vhdx comp:page

The general idea is the verb, standard and custom request headers, canonical headers, canonical resource and query are presented as a newline delimited string. This string is encoded using the HMAC256 algorithm with the storage account key. This base64 encoded string is used for crafting the Authorization header. The Authorization header is passed with the other headers used to sign the request. If the server is able to match the signature, the request is authenticated.

Putting this in some PoSh

First things first, we need to generate the string to sign. This function will take arguments for the desired HTTP request (URI, Verb, Query, Headers) parameters and create the previously described string.

Function GetTokenStringToSign
{
    [CmdletBinding()]     
    param
    (
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [ValidateSet('GET','PUT','DELETE')]
        [string]$Verb="GET",
        [Parameter(Mandatory=$true,ValueFromPipelineByPropertyName = $true)]
        [System.Uri]$Resource,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [long]$ContentLength,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [String]$ContentLanguage,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [String]$ContentEncoding,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [String]$ContentType,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [String]$ContentMD5,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [long]$RangeStart,
        [Parameter(Mandatory = $false,ValueFromPipelineByPropertyName = $true)]
        [long]$RangeEnd,[Parameter(Mandatory = $true,ValueFromPipelineByPropertyName = $true)]
        [System.Collections.IDictionary]$Headers
    )

    $ResourceBase=($Resource.Host.Split('.') | Select-Object -First 1).TrimEnd("`0")
    $ResourcePath=$Resource.LocalPath.TrimStart('/').TrimEnd("`0")
    $LengthString=[String]::Empty
    $Range=[String]::Empty
    if($ContentLength -gt 0){$LengthString="$ContentLength"}
    if($RangeEnd -gt 0){$Range="bytes=$($RangeStart)-$($RangeEnd-1)"}

    $SigningPieces = @($Verb, $ContentEncoding,$ContentLanguage, $LengthString,$ContentMD5, $ContentType, [String]::Empty, [String]::Empty, [String]::Empty, [String]::Empty, [String]::Empty, $Range)
    foreach ($item in $Headers.Keys)
    {
        $SigningPieces+="$($item):$($Headers[$item])"
    }
    $SigningPieces+="/$ResourceBase/$ResourcePath"

    if ([String]::IsNullOrEmpty($Resource.Query) -eq $false)
    {
        $QueryResources=@{}
        $QueryParams=$Resource.Query.Substring(1).Split('&')
        foreach ($QueryParam in $QueryParams)
        {
            $ItemPieces=$QueryParam.Split('=')
            $ItemKey = ($ItemPieces|Select-Object -First 1).TrimEnd("`0")
            $ItemValue = ($ItemPieces|Select-Object -Last 1).TrimEnd("`0")
            if($QueryResources.ContainsKey($ItemKey))
            { 
                $QueryResources[$ItemKey] = "$($QueryResources[$ItemKey]),$ItemValue"    
            }
            else
            {
                $QueryResources.Add($ItemKey, $ItemValue)
            }
        }
        $Sorted=$QueryResources.Keys|Sort-Object
        foreach ($QueryKey in $Sorted)
        {
            $SigningPieces += "$($QueryKey):$($QueryResources[$QueryKey])"
        }
    }

    $StringToSign = [String]::Join("`n",$SigningPieces)
    Write-Output $StringToSign 
}

Once we have the signature, it is a simple step create the required HMACSHA256 Hash using the storage account key. The following function takes the two arguments and returns the encoded signature.

Function EncodeStorageRequest
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory = $true,ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)]
        [String[]]$StringToSign,
        [Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true)]
        [String]$SigningKey
    )
    PROCESS
    {         
        foreach ($item in $StringToSign)
        {
            $KeyBytes = [System.Convert]::FromBase64String($SigningKey)
            $HMAC = New-Object System.Security.Cryptography.HMACSHA256
            $HMAC.Key = $KeyBytes
            $UnsignedBytes = [System.Text.Encoding]::UTF8.GetBytes($item)
            $KeyHash = $HMAC.ComputeHash($UnsignedBytes)
            $SignedString=[System.Convert]::ToBase64String($KeyHash)
            Write-Output $SignedString 
        }     
    }
}

Now that we have a signature it is time to pass it on to the storage service API, for the following examples we will focus on BLOB. Let's return to the first example, retrieving a list of the BLOBs in the certificates container of the nosaashere storage account. This only requires the date and version API headers. This request would take the format:

GET https://nosaashere.blob.core.windows.net/certificates?restype=container&amp;comp=list x-ms-date:Mon, 08 May 2017 23:28:20 GMT x-ms-version:2016-05-31

To create the signature we can use the above function.

$StorageAccountName='nosaashere'
$ContainerName='certificates'
$AccessKey="WMTyrXNLHL+DF4Gwn1HgqMrpl3s8Zp7ttUevo0+KN2adpByHaYhX4OBY7fLNyzw5IItopGDAr8iQDxrhoHHiRg=="
$BlobContainerUri="https://$StorageAccountName.blob.core.windows.net/$ContainerName?restype=container&comp=list"
$BlobHeaders= @{
    "x-ms-date"=[DateTime]::UtcNow.ToString('R');
     "x-ms-version"='2016-05-31'; 
}
$UnsignedSignature=GetTokenStringToSign -Verb GET -Resource $BlobContainerUri -AccessKey $AccessKey -Headers $BlobHeaders $StorageSignature=EncodeStorageRequest -StringToSign $UnsignedSignature -SigningKey $SigningKey 
#Now we should have a 'token' for our actual request. 
$BlobHeaders.Add('Authorization',"SharedKey $($StorageAccountName):$($StorageSignature)") 
$Result=Invoke-RestMethod -Uri $Uri -Headers $BlobHeaders –UseBasicParsing

If you make your call without using the -OutFile parameter you will find a weird looking string rather than the nice friendly XmlDocument you were expecting.

ï»¿<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://nosaashere.blob.core.windows.net/" ContainerName="certificates">
    <Blobs>
        <Blob>
            <Name>azurestackroot.as01.cer</Name>
            <Properties>
                <Last-Modified>Fri, 05 May 2017 20:31:33 GMT</Last-Modified>
                <Etag>0x8D493F5B8410E96</Etag>
                <Content-Length>1001</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-MD5>O2/fcFtzb9R6alGEgXDZKA==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>false</ServerEncrypted>
            </Properties>
        </Blob>
        <Blob>
            <Name>azurestackroot.as02.cer</Name>
            <Properties>
                <Last-Modified>Wed, 03 May 2017 22:54:49 GMT</Last-Modified>
                <Etag>0x8D4927767174A24</Etag>
                <Content-Length>1001</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-MD5>arONICHXLfRUr61IH/XHbw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>false</ServerEncrypted>
            </Properties>
        </Blob>
        <Blob>
            <Name>azurestackroot.as03.cer</Name>
            <Properties>
                <Last-Modified>Wed, 15 Mar 2017 19:43:50 GMT</Last-Modified>
                <Etag>0x8D46BDB9AB84CFD</Etag>
                <Content-Length>1001</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-MD5>sZZ30o/oMO57VMfVR7ZBGg==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>false</ServerEncrypted>
            </Properties>
        </Blob>
        <Blob>
            <Name>azurestackroot.as04.cer</Name>
            <Properties>
                <Last-Modified>Wed, 26 Apr 2017 22:45:41 GMT</Last-Modified>
                <Etag>0x8D48CF5F7534F4B</Etag>
                <Content-Length>1001</Content-Length>
                <Content-Type>application/octet-stream</Content-Type>
                <Content-Encoding />
                <Content-Language />
                <Content-MD5>rnkI6VPz9i1pXOick4qDSw==</Content-MD5>
                <Cache-Control />
                <Content-Disposition />
                <BlobType>BlockBlob</BlobType>
                <LeaseStatus>unlocked</LeaseStatus>
                <LeaseState>available</LeaseState>
                <ServerEncrypted>false</ServerEncrypted>
            </Properties>
        </Blob>
    </Blobs>
    <NextMarker />
</EnumerationResults>

What, pray tell is this ï»¿ ? In a weird confluence of events there is a long standing 'issue' with the Invoke-RestMethod and Invoke-WebRequest Cmdlets with the UTF-8 BOM. Luckily, .Net has lots of support for this stuff. Generally, most people just use the OutFile parameter and pipe it along to the Get-Content Cmdlet. If you are like me, we'll look for the UTF-8 preamble and strip it from the string.

$UTF8ByteOrderMark=[System.Text.Encoding]::Default.GetString([System.Text.Encoding]::UTF8.GetPreamble())
if($Result.StartsWith($UTF8ByteOrderMark,[System.StringComparison]::Ordinal))
{
    $Result=$Result.Remove(0,$UTF8ByteOrderMark.Length)
}
[Xml]$EnumerationResult=$Result

Now you'll see something you should be able to work with:

PS C:\Users\chris> $ResultXml.EnumerationResults
ServiceEndpoint                           ContainerName Blobs NextMarker
---------------                           ------------- ----- ----------
https://nosaashere.blob.core.windows.net/ certificates Blobs
PS C:\Users\chris> $ResultXml.EnumerationResults.Blobs.Blob
Name                    Properties
----                    ----------
azurestackroot.as01.cer Properties 
azurestackroot.as02.cer Properties 
azurestackroot.as03.cer Properties
azurestackroot.as04.cer Properties

All storage service requests return a good deal of information in the response headers. Enumeration style operations , like the previous example return the relevant data in the response body. Many operations, like retrieving container or BLOB metadata return only relevant data in the response headers. Let’s modify our previous request, noting the change in the query parameter. You will also need to use the Invoke-WebRequest Cmdlet (or your other favorite method) so that you can access the response headers.

$BlobContainerUri="https://$StorageAccountName.blob.core.windows.net/$ContainerName?restype=container&comp=metadata"
$BlobHeaders= @{ "x-ms-date"=[DateTime]::UtcNow.ToString('R'); "x-ms-version"='2016-05-31'; }
$UnsignedSignature=GetTokenStringToSign -Verb GET -Resource $BlobContainerUri `
    -AccessKey $AccessKey -Headers $BlobHeaders $StorageSignature=EncodeStorageRequest `
    -StringToSign $UnsignedSignature -SigningKey $SigningKey
$BlobHeaders.Add('Authorization',"SharedKey $($StorageAccountName):$($StorageSignature)")
$Response=Invoke-WebRequest -Uri $Uri -Headers $BlobHeaders –UseBasicParsing
$ContainerMetadata=$Response.Headers

We should have the resulting metadata key-value pairs present in the form x-ms-meta-<Key Name>.

C:\Users\chris> $ContainerMetaData
Key                      Value
---                      ----- 
Transfer-Encoding        chunked
x-ms-request-id          5f15423e-0001-003d-066d-ca0167000000
x-ms-version             2016-05-31
x-ms-meta-dispo          12345
x-ms-meta-stuff          test
Date                     Thu, 11 May 2017 15:41:16 GMT
ETag                     "0x8D4954F4245F500"
Last-Modified            Sun, 07 May 2017 13:45:01 GMT
Server                   Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0

Where to go from here?

With the authentication scheme in hand, you can now access the all of the storage service. This includes creating snapshots, uploading and downloading files. If you are not inclined to do things the hard way, feel free to check out a module supporting most of the BLOB service functionality on the Powershell Gallery or GitHub.

Topic Search

Azure Field Notes

Securing TLS in WAC (Windows Admin Center) https://t.co/klDc7J7R4G
Jan 27, 2023, 1:11 PM

Posts by Date