Terraform and WSL2 issue

Here’s a quick note on an issue that I encountered today (plus it seems, many other people).

I went to run a Terraform workflow on my system via WSL2, but I cam across a number of problems.

First, was that I couldn’t obtain the State that was stored in an Azure Storage account container. Previously, I used the following config:

backend "azurerm" {
    resource_group_name  = ""
    storage_account_name = ""
    container_name       = "terraform-backend"
    key                  = ""
 }

At runtime, I would specify the values like the example below.

export TF_CLI_ARGS_init="-backend-config=\"storage_account_name=${TERRAFORM_STATE_CONTAINER_NAME}\" -backend-config=\"resource_group_name=${RESOURCE_GROUP_NAME}\" -backend-config=\"access_key=${STG_KEY}\""

However, today, that didn’t work as it just stalled trying to connect to the storage container.

I thought it was something wrong with my credentials, so for troubleshooting purposes, I added the storage account key to see if that made a difference

backend "azurerm" {
    resource_group_name  = ""
    storage_account_name = ""
    container_name       = "terraform-backend"
    key                  = ""
    access_key           = ""
}

I added the primary storage key and lo and behold, this time, it worked.

Strange, as I hadn’t updated the terraform cli or providers.

The next problem I saw was that when I tried to run

terraform plan

it would not complete, seemingly freezing. To troubleshoot this, I ran

export TF_LOG="TRACE"

before running the plan to tell me what was happening in the background.

This in turn produces a verbose output, but something that did catch my was this:

View fullsize

Strange. I know I have internet connectivity and I could certainly connect to Azure using az cli, so I did some Goole-fu and found the following: https://github.com/microsoft/WSL/issues/8022

It was exactly the same problem I had encountered.

Applying the fix https://github.com/microsoft/WSL/issues/5420#issuecomment-646479747 worked for me and persisted beyond a reboot.

(run the code below in your WSL2 instance)

sudo rm /etc/resolv.conf
sudo bash -c 'echo "nameserver 8.8.8.8" > /etc/resolv.conf'
sudo bash -c 'echo "[network]" > /etc/wsl.conf'
sudo bash -c 'echo "generateResolvConf = false" >> /etc/wsl.conf'
sudo chattr +i /etc/resolv.conf

It appears to have occurred in the latest Windows update and affects WSL2. It only appears to affect Go / Terraform as far as I can tell.

Hopefully this will help anyone having a similar issue until the Go provider is fixed.

Azure Bastion undocumented requirement gotcha

Just a quick post to highlight an undocumented requirement for Azure Bastion that I came across when deploying a Landing Zone.

I’m creating a new landing zone for a client and we’re using Azure Bastion for secure access to IaaS VM’s. I decided to create the resource in a separate resource group than the Virtual Network as it was uncertain whether this was going to be required long term or not. There’s nothing in the current documentation that indicates that it isn’t possible, so I tried to deploy.

After a few minutes, it failed:

View fullsize

Here’s the less than helpful error:

View fullsize

No matter what I tried (Portal, Terraform, Azure CLI), the same occurred.

Upon speaking to Azure Support, this is a known issue and the mitigation is to deploy the Bastion host within the same Resource Group as the Virtual Network that it is trying to connect to.

I’ve experienced the same when deploying API Management in Azure, but at least the errors from ARM are meaningful and pointed me in the right direction.

Hopefully if you come across the same, and the problem isn’t resolved, this will help you out.

Capturing and using API queries from Azure in PowerShell with Fiddler

Similar New Content: Using Developer Tools to get the Payload to Create Azure Budget Alerts for Action Groups (New-AzConsumptionBudget) — Crying Cloud

This is a walkthrough for using Fiddler to capture traffic to Azure from a browser and writing and running that query in PowerShell.  I wrote this because I don't like posts that skip over a key step and explain the entire thing with a wave of the hand.  Although this article stands on it own, it is a key step in another series.

Install & Configure Fiddler

https://www.telerik.com/fiddler. We need to decrypt traffic from Azure.  Basically, you're letting Fiddler get in the middle of the conversation you're having with Azure and take a look at the traffic.  After installation select, Tools -> Options, select capture and decrypt HTTPS traffic.

You need to close and reopen Fiddler.  From the file menu, you can select start or stop, to capture internet traffic. Ensure capture is on and then refresh Azure page you want to query.  In this case, I want to capture data from the cost analysis page for a resource group.  This is easier when you close down most browser windows except the one you want to investigate.  You can also apply filters and capture limitations, but I'll let you figure that out.

Capture Your Query

I want to capture the cost for a resource group with daily totals so we can capture the cost over the month based on resource group tagging.  Browse to a resource group and select cost analysis with capture traffic.

The next part you'll just need to search through the queries and look for what you're after.  Select JSON in the response to see the data returned.

In the above results we can see the rows of cost data in the JSON response page, however, the other record in the row is the resource type, not the date.

This looks better, the columns JSON field shows the Cost, Date, and Currency and we can even see some rows with the right data, so we have the query. Now to create this in PowerShell.

Create Query in PowerShell

First, grab the header and then create a few parameters.  Note this is a POST command.

Raw Header

 
POST /subscriptions/11111111-4444-8888-9999-222222222222/YourResourceGroupName/azurestackproduction/providers/Microsoft.CostManagement/query?api-version=2018-08-31&$top=40000 HTTP/1.1

Converted

$SubscriptionGUID = '11111111-4444-8888-9999-222222222222'  $ResourceGroupName = 'YourResourceGroupName' $usageUri =  "https://management.azure.com/subscriptions/$SubscriptionGUID/resourceGroups/$ResourceGroupName/providers/Microsoft.CostManagement/query?api-version=2018-08-31"

We need to create the JSON object that is passed with the POST. Shown above is what we need to recreate.

Select Raw and capture the text in the brackets. This will take a little bit of effort to convert into a PowerShell JSON object with variables.

• commas , become semi colons ;

• the { needs a @ in front of it @{

• colons : need =

RAW

 
{"type":"Usage","timeframe":"Custom","timePeriod":{"from":"2018-10-01T00:00:00+00:00","to":"2018-10-31T23:59:59+00:00"},"dataSet":{"granularity":"Daily","aggregation":{"totalCost":{"name":"PreTaxCost","function":"Sum"}},"sorting":[{"direction":"ascending","name":"UsageDate"}]}}

Converted

$year =(get-date).year $month =(get-date).Month $DaysInMonth= [DateTime]::DaysInMonth($year, $month ) $Body = @{"type"="Usage";"timeframe"="Custom";"timePeriod"=@{"from"="$($year)-$($month)-01T00:00:00+00:00";"to"="$($year)-$($month)-$($DaysInMonth)T23:59:59+00:00"};"dataSet"=@{"granularity"="Daily";"aggregation"=@{"totalCost"=@{"name"="PreTaxCost";"function"="Sum"}};"sorting"=@(@{"direction"="ascending";"name"="UsageDate"})}}

BearerToken

To access this data since we aren't logged in with PowerShell you need a bearer token.  Luckily someone has written a helpful query to capture the bearer token from your existing session. https://gallery.technet.microsoft.com/scriptcenter/Easily-obtain-AccessToken-3ba6e593. 

function Get-AzureRmCachedAccessToken() {   $ErrorActionPreference = 'Stop'

  if(-not (Get-Module AzureRm.Profile)) {     Import-Module AzureRm.Profile   }   $azureRmProfileModuleVersion = (Get-Module AzureRm.Profile).Version   # refactoring performed in AzureRm.Profile v3.0 or later   if($azureRmProfileModuleVersion.Major -ge 3) {     $azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile     if(-not $azureRmProfile.Accounts.Count) {       Write-Error "Ensure you have logged in before calling this function."         }   } else {     # AzureRm.Profile < v3.0     $azureRmProfile = [Microsoft.WindowsAzure.Commands.Common.AzureRmProfileProvider]::Instance.Profile     if(-not $azureRmProfile.Context.Account.Count) {       Write-Error "Ensure you have logged in before calling this function."         }   }

  $currentAzureContext = Get-AzureRmContext   $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile)   Write-Debug ("Getting access token for tenant" + $currentAzureContext.Subscription.TenantId)   $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId)   $token.AccessToken }

If we include this function in our code and write a few more lines we are ready to start putting it all together. We create the headers sections, we use invoke-restmethod with POST we pass the body which must be converted with depth 100 otherwise data gets chopped out.

$token = Get-AzureRmCachedAccessToken  $headers = @{"authorization"="bearer $token"} 

$results = Invoke-RestMethod $usageUri -Headers $headers -ContentType "application/json" -Method Post -Body ($body | ConvertTo-Json -Depth 100) 

Final Script

$SubscriptionGUID = '11111111-4444-8888-9999-222222222222'  $ResourceGroupName = 'YourResourceGroupName'

function Get-AzureRmCachedAccessToken() {   $ErrorActionPreference = 'Stop'

  if(-not (Get-Module AzureRm.Profile)) {     Import-Module AzureRm.Profile   }   $azureRmProfileModuleVersion = (Get-Module AzureRm.Profile).Version   # refactoring performed in AzureRm.Profile v3.0 or later   if($azureRmProfileModuleVersion.Major -ge 3) {     $azureRmProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile     if(-not $azureRmProfile.Accounts.Count) {       Write-Error "Ensure you have logged in before calling this function."         }   } else {     # AzureRm.Profile < v3.0     $azureRmProfile = [Microsoft.WindowsAzure.Commands.Common.AzureRmProfileProvider]::Instance.Profile     if(-not $azureRmProfile.Context.Account.Count) {       Write-Error "Ensure you have logged in before calling this function."         }   }

  $currentAzureContext = Get-AzureRmContext   $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azureRmProfile)   Write-Debug ("Getting access token for tenant" + $currentAzureContext.Subscription.TenantId)   $token = $profileClient.AcquireAccessToken($currentAzureContext.Subscription.TenantId)   $token.AccessToken }

$year =(get-date).year $month =(get-date).Month $DaysInMonth= [DateTime]::DaysInMonth($year, $month )

$token =  Get-AzureRmCachedAccessToken $headers = @{"authorization"="bearer $token"} $Body =  @{"type"="Usage";"timeframe"="Custom";"timePeriod"=@{"from"="$($year)-$($month)-01T00:00:00+00:00";"to"="$($year)-$($month)-$($DaysInMonth)T23:59:59+00:00"};"dataSet"=@{"granularity"="Daily";"aggregation"=@{"totalCost"=@{"name"="PreTaxCost";"function"="Sum"}};"sorting"=@(@{"direction"="ascending";"name"="UsageDate"})}} 

$usageUri = "https://management.azure.com/subscriptions/$SubscriptionGUID/resourceGroups/$ResourceGroupName/providers/Microsoft.CostManagement/query?api-version=2018-08-31"    $results = Invoke-RestMethod $usageUri -Headers $headers -ContentType "application/json" -Method Post -Body ($body | ConvertTo-Json -Depth 100) 

$results.properties.columns $results.properties.rows

Results

This shows the two output selected columns and rows

Good luck creating your own queries, I hope you found this helpful.

You can find another similar article by Microsoft here

Resource Tagging Best Practices Applied (Part 2 – Enforcement)

The Real Problem

This post is following on from part 1 about resource tagging on resource groups where we setup azure policies to look for the existence of resource tags on resource groups.  While this is helpful to understand the scale of the problem, the real problem is getting people to tag their resource groups when they create them.  I work with a bunch of misfits and mavericks and while all brilliant in their own right, asking them to do anything as simple as tagging their stuff is about as futile as yelling at the rain to stop.

The Solution

Since asking failed, let's try telling them.  Same as in part one let's assign a policy that will force tags to be applied during object creation.  You can set the tag value to the text wildcard *

While this will work 100% of the time, it does come along with a few issues.  This list is by no means exhaustive and I will update it when and if we find more.  If you have tried or are trying this and find any other issues arising from enforcing tags on resource groups, please comment and I can explore and add the content to this post.

Can't use the Portal

At the time of writing this, unfortunately, Azure does not ask you to add tags during the creation of Resource groups through the UI so you simply get an error.

You have to use PowerShell or ARM templates to create resource groups.

New-AzureRmResourceGroup -Name "Blahdedah" -Location "WestUS" -Verbose -Force -Tag @{Owner="matt quickenden"; Solution="IoT Testing"}

Adding a template to Azure

So you're thinking you could upload a template with parameters for tags to Azure Template so you could keep a UI experience? 

{     "$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",     "contentVersion": "1.0.0.1",     "parameters": {         "ResourceGroupName": {             "type": "string"         },         "ResourceGroupLocation": {             "type": "string"         }, 		"OwnerTag": { 			"type": "string" 		}, 		"SolutionTag": { 			"type": "string" 		}     },     "variables": { 	 "tags": {       "Owner": "[parameters('OwnerTag')]",       "Solution": "[parameters('SolutionTag')]"     } 	},     "resources": [         {             "type": "Microsoft.Resources/resourceGroups",             "apiVersion": "2018-05-01",             "location": "[parameters('ResourceGroupLocation')]",             "name": "[parameters('ResourceGroupName')]",             "properties": {}, 			"tags": "[variables('tags')]"         }     ],     "outputs": {} }

Close enough, you could limit the location to actual Azure locations etc, but let's check if it works.

Interestingly, Azure creates a resource group first before trying to execute your code.  This could work for creating a blank template using ARM but the PowerShell is probably easier or just including the tags in your main ARM template.

Visual Studio Deployments

Deploying quickstart templates from visual studio fails.

the workaround is to open the Deploy-AzureResourceGroup.ps1 script inside the project. Scroll down to the line that starts with New-AzureRmResourceGroup. It should be around line 93.

Edit that line so your tags are added at deployment. That line should look something like this after you have edited it:

New-AzureRmResourceGroup -Name $ResourceGroupName -Location $ResourceGroupLocation -Tag @{Owner="harry potter"; Solution="Deploy with tags "} -Verbose -Force

Azure Backup

Using Azure backup vault to create a backup of a VM.  If you have it already set up it seems to be just fine, however, when you attempt to create a new one backup it seems to fail.

Taking a look at the activity log reveals 'Update resource group' failed

cutting through the rest we can find the status error message

"statusMessage": "{\"error\":{\"code\":\"RequestDisallowedByPolicy\",\"target\":\"AzureBackupRG_westus_1\",\"message\":\"Resource 'AzureBackupRG_westus_1' was disallowed by policy. Policy identifiers: '[{\\\"policyAssignment\\\":{\\\"name\\\":\\\"Enforce Resource Group Tag Like Value SOLUTION Pattern\\\",\\\"id\\\":\\\"/subscriptions/....

Adding an exclusion to the enforcement policy for the resource group seems to have done the trick.  New backups to this backup vault can be created and continue to run without any issues.

Next Steps

While this enforcement has created some problems, there aren't any show stoppers at the moment and if it really is an issue, for a particular use case or project if you can't simply add an exclusion you can disable the policy temporarily and re-enable it for a month. Some deployments might get through without tags but we can hunt those people down through the activity logs.  This is more a catch-all tool so I still consider this useful and still functional so we will proceed.

Next up we will take a look into getting at the data and trying to get closer to the ultimate goal of putting data in an email targeted at the resource group owners.

Resource Tagging Best Practices Applied (Part 1 - Auditing)

Our most popular blog post was about resource tagging best practices. I thought I would follow up that post with some real-world application of tagging best practices in our own environment with the explicit purpose of tracking down Azure spend and getting that spend information into people's inboxes so they can take action to reduce costs.

The Environment

• Our group pays one bill and we don't charge back the cost of Azure spend, so we technically don't have a need to track charge codes. A person is responsible for objects and those objects are part of a solution or project so we have two attributes we are interested in capturing.

• We have two subscriptions to separate our environments so we don't need an environment tag. The two environments are;

  • Critical Infrastructure

  • Labs

  • We are using only two tags at a resource group level

    • Owner

    • Solution

Azure Policy & Policy Definitions

Azure Policy has a number of built-in policies, however, it doesn't have one for Auditing Resource Tags.  Thankfully, we have a quick win, https://github.com/Azure/azure-policy/tree/master/samples/ResourceGroup/audit-resourceGroup-tags.  You will need to be a subscription owner to create this policy definition.

$definition = New-AzureRmPolicyDefinition -Name "audit-resourceGroup-tags" -DisplayName "Audit resource groups missing tags" -description "Audit resource groups that doesn't have particular tag" -Policy 'https://raw.githubusercontent.com/Azure/azure-policy/master/samples/ResourceGroup/audit-resourceGroup-tags/azurepolicy.rules.json' -Parameter 'https://raw.githubusercontent.com/Azure/azure-policy/master/samples/ResourceGroup/audit-resourceGroup-tags/azurepolicy.parameters.json' -Mode All $definition

Let's do the next step through the UI.  Go to Policy, Assignments, Assign Policy, Select your Subscription.  You can also select resource groups for exclusion (more on that later) for audit purposes I would like to target the entire subscription.

Next, select the Policy Definition, search for word 'tag'.  Here we can see the built-in definitions and the custom definition we have just uploaded.

Policy Assignment

Once selected, you can complete the remaining fields.  We need to create policy assignments for auditing Owner, Solution tags for both subscriptions.

Once complete you should be able to see the following

Compliance

Which if we select compliance we can see a summary of all the policies

If we select one of the audits, we can see the items that have failed to match the assigned policy, that is resource groups do not have the Owner resource tag.

While this helps find resource groups that are not tagged, the problem is that if someone spins up some resources and destroys them that usage data has no tags associated with it and therefore we can't track who provisioned it.  I was using the Activity log to try and find who was working with the resources or had created it.

Defining an Initiative

Alternatively, you can combine these policies into an Initiative, basically a group of policies.

In this case I have defined the values in the initiative, but you can also use parameters.  You then have to assign the initiative to a subscription

Here you can see there are two policies are part of this initiative

and then the compliance is summarized

Next Steps

Enforcing Owner and Solution resource tags on resource groups.