Creating MAAS workload annotations for Kubernetes using kubectl output
Trying to manually keep track of servers and versions is challenging. MAAS has a workload annotation feature that allows you to create name=value using command line calls using MASS cli. I wanted to explore creating a script to pull data from kubectl and create workload annotations for machines in a Kubernetes cluster.
For this exercise, I intend to execute the script from a MAAS rack controller, on which I have installed kubectl and PowerShell. In the lab, there are three microk8s clusters. The first job was to collect the config files from each cluster and combine them into a single config so I can use contexts from within the kubectl commands. Using the command ‘kubectl config view --raw > kn.conf’ I created one file for each k8s cluster on the region controller in directory ~/.kube/clusters.
mkdir ~/.kube/clusters
mv /path/k1.conf ~/.kube/clusters
mv /path/k2.conf ~/.kube/clusters
export KUBECONFIG=$(find ~/.kube/clusters -type f | sed ':a;N;s/\n/:/;ba')
kubectl config get-clusters
kubectl config view --flatten > ~/.kube/configThere are various methods to combine and clean up k8s cluster contexts. I had to clean up the file to create unique cluster names as well as user token names. I manually edited the ~/.kube/config combined file and tweaked these settings.
This is the script I created. It does require that you have a MAAS profile created already.
# tag-maasworkloadannotations.ps1
[CmdletBinding()]
param (
$k8scontext ,
$maasProfile
)
write-verbose "executing tag kubernetes workload annotation for MAAS"
$LastUpdated = Get-Date
# Kubectl contexts
Write-Verbose "Retrieving Kubernetes cluster details from context $k8scontext"
$kcontexts = kubectl config view -o json | convertfrom-json
Write-Verbose "$(($kcontexts.contexts | Measure-Object).count) kube contexts found"
$kcontext = $kcontexts.contexts | Where-Object {$_.name -eq $k8scontext}
$kversion = kubectl version --context $k8scontext -o json | convertfrom-json
# K8s nodes
Write-Verbose "Retrieving Kubernetes nodes from context $k8scontext"
$nodes = kubectl get nodes --context $k8scontext -o json | convertfrom-json
Write-Verbose "$(($nodes.items | Measure-Object).count) k8s nodes found"
# Maas machines
Write-Verbose "Retrieving machines from MAAS using profile $maasProfile"
$machines = maas $maasProfile machines read | convertfrom-json
Write-Verbose "$($machines.count) maas machines found"
$powerParams = maas $maasProfile machines power-parameters | convertfrom-json
# Build Annotations
Write-Verbose "Building workload annotation records"
$WorkloadAnnontations = @()
foreach ($node in $nodes.items) {
$WorkloadAnnontation = @{}
$WARecord = '' | select-object systemid, hostname, WA
$machine = $machines | Where-Object {$_.hostname -eq $node.metadata.name}
if ($machine -ne ""){
$WARecord.systemid = $machine.system_id
$WARecord.hostname = $machine.hostname
#$WorkloadAnnontation.add("osImage", $node.status.nodeInfo.osImage)
#$WorkloadAnnontation.add("systemUUID", $node.status.nodeInfo.systemUUID)
#$WorkloadAnnontation.add("machineID", $node.status.nodeInfo.machineID)
$WorkloadAnnontation.add("k8scluster", $kcontext.context.cluster )
$WorkloadAnnontation.add("buildDate", $kversion.serverVersion.buildDate)
$WorkloadAnnontation.add("containerRuntimeVersion", $node.status.nodeInfo.containerRuntimeVersion)
$WorkloadAnnontation.add("kernelVersion", $node.status.nodeInfo.kernelVersion)
$WorkloadAnnontation.add("kubeProxyVersion", $node.status.nodeInfo.kubeProxyVersion)
$WorkloadAnnontation.add("kubeletVersion", $node.status.nodeInfo.kubeletVersion)
$WorkloadAnnontation.add("outofband", $powerParams.$($machine.system_id).power_address)
$WorkloadAnnontation.add("AnnotationUpdated",$LastUpdated)
if ($node.metadata.labels.'node.kubernetes.io/microk8s-controlplane' -eq "microk8s-controlplane")
{$WorkloadAnnontation.add("nodeType", 'Master')}
if ($node.metadata.labels.'node.kubernetes.io/microk8s-worker' -eq "microk8s-worker")
{$WorkloadAnnontation.add("nodeType", 'Worker')}
$WARecord.wa = $WorkloadAnnontation
}
$WorkloadAnnontations += $WARecord
}
# publish workload annotations
$i = 1
$c = $($WorkloadAnnontations.count)
Write-Verbose "Publishing $c workload annotation records"
foreach ($WA in $WorkloadAnnontations){
$KeyValueData = ""
foreach ($key in $WA.wa.keys) {
$KeyValueData += "$($key)='$($wa.wa.($key))' "
}
Write-Verbose "[$i/$c] Building command for $($WA.hostname)"
$execmd = "maas $maasProfile machine set-workload-annotations $($WA.systemID) $KeyValueData > /dev/null 2>&1"
write-debug $execmd
Invoke-Expression $execmd
Write-Verbose "[$i/$c] Command executed for $($WA.hostname)"
$i++
}
$RunTime = New-TimeSpan -Start $LastUpdated -End (get-date)
$ExecutionTime = "Execution time was {0} hours, {1} minutes, {2} seconds and {3} milliseconds." -f $RunTime.Hours, $RunTime.Minutes, $RunTime.Seconds, $RunTime.Milliseconds
write-verbose $ExecutionTimeUsing the [CmdletBinding()] allows me to leverage verbose and debug options.
./tag-maasworkloadannotations.ps1 -k8scontext k8sdemo -maasprofile mquick -Verbose -debug./tag-maasworkloadannotations.ps1 -k8scontext microk8s -maasprofile mquick -Verbose
This took 12 minutes but could probably create streamlined script with PowerShell jobs
which can allow combinations of filters on the workload annotations
There is a lot more possible here, this was helpful for me while consolidating microk8s clusters and making sure I wasn’t releasing machines that were in use in a cluster if I had tagged them incorrectly in MAAS.
External Reference
Using multiple kubeconfig files and how to merge to a single – Oueta
Azure Arc Connected K8s Billing usage workbook error - 'union' operator: Failed to resolve table expression named 'ContainerLogV2'
Using ‘Container Insights’ for Kubernetes can get expensive, I recently saw a customer creating 19TB of logs a month from logging container insights from an AKS cluster.
You may want to check what logs you are capturing as suggested through Microsoft guides Monitoring cost for Container insights - Azure Monitor | Microsoft Docs which suggests one of the first steps to check billing usage.
As of writing this if you enable container insights and try to view Data Usage you get an error. ‘union' operator: Failed to resolve table expression named 'ContainerLogV2’. ContainerLogV2 is in preview or if you are just aren’t ready to deploy that, here is how to fix the report manually. I also have included the Full report at the end.
However, depending on when you deploy this and how you deployed insight metrics you may hit this error.
As of writing this, ContainerLogV2 is in preview and you can follow this guide to deploy it. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs. If you want to gain some quick insight into your usage you can edit the workbook. We just need to get through the maze of edits to get to the underlying query.
Select ‘Azure Arc’ services
select ‘Kubernetes clusters’ under infrastructure
select your K8s cluster
select ‘Workbooks’ under Monitoring
select ‘Container Insights Usage’ workbook
you can either edit this one or ‘Save As’ and create a new name. I will create a copy.
Open your chosen workbook
you can now edit the group item.
by Namespace
Editing the Namespace selection Dropdown parameter query
Hopefully, you get the gist of the process. There are a few more queries to edit.
And
This will now work for Data Usage. This is likely a temporary solution until ContainerLogV2 comes out of preview. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs
Gallery Template code
{
"version": "Notebook/1.0",
"items": [
{
"type": 9,
"content": {
"version": "KqlParameterItem/1.0",
"crossComponentResources": [
"{resource}"
],
"parameters": [
{
"id": "670aac26-0ffe-4f29-81c2-d48911bc64b6",
"version": "KqlParameterItem/1.0",
"name": "timeRange",
"label": "Time Range",
"type": 4,
"description": "Select time-range for data selection",
"isRequired": true,
"value": {
"durationMs": 21600000
},
"typeSettings": {
"selectableValues": [
{
"durationMs": 300000
},
{
"durationMs": 900000
},
{
"durationMs": 3600000
},
{
"durationMs": 14400000
},
{
"durationMs": 43200000
},
{
"durationMs": 86400000
},
{
"durationMs": 172800000
},
{
"durationMs": 259200000
},
{
"durationMs": 604800000
},
{
"durationMs": 1209600000
},
{
"durationMs": 2419200000
},
{
"durationMs": 2592000000
},
{
"durationMs": 5184000000
},
{
"durationMs": 7776000000
}
],
"allowCustom": true
}
},
{
"id": "bfc96857-81df-4f0d-b958-81f96d28ddeb",
"version": "KqlParameterItem/1.0",
"name": "resource",
"type": 5,
"isRequired": true,
"isHiddenWhenLocked": true,
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 21600000
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1"
},
{
"id": "8d48ec94-fde6-487c-98bf-f1295f5d8b81",
"version": "KqlParameterItem/1.0",
"name": "resourceType",
"type": 7,
"description": "Resource type of resource",
"isRequired": true,
"query": "{\"version\":\"1.0.0\",\"content\":\"\\\"{resource:resourcetype}\\\"\",\"transformers\":null}",
"isHiddenWhenLocked": true,
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 21600000
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1",
"queryType": 8
},
{
"id": "1b826776-ab99-45ab-86db-cced05e8b36d",
"version": "KqlParameterItem/1.0",
"name": "clusterId",
"type": 1,
"description": "Used to identify the cluster name when the cluster type is AKS Engine",
"isHiddenWhenLocked": true,
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange"
},
{
"id": "67227c35-eab8-4518-9212-1c3c3d564b20",
"version": "KqlParameterItem/1.0",
"name": "masterNodeExists",
"type": 1,
"query": "let MissingTable = view () { print isMissing=1 };\r\nlet masterNodeExists = toscalar(\r\nunion isfuzzy=true MissingTable, (\r\nAzureDiagnostics \r\n| getschema \r\n| summarize c=count() \r\n| project isMissing=iff(c > 0, 0, 1)\r\n) \r\n| top 1 by isMissing asc\r\n);\r\nprint(iif(masterNodeExists == 0, 'yes', 'no'))\r\n",
"crossComponentResources": [
"{resource}"
],
"isHiddenWhenLocked": true,
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}"
}
],
"style": "pills",
"queryType": 0,
"resourceType": "{resourceType}"
},
"name": "pills"
},
{
"type": 1,
"content": {
"json": "Please note that the Container Insights Usage workbook for AKS Engine clusters shows your billable data usage for your cluster's entire workspace ({resource:name}), and not just the cluster itself ({clusterId}). ",
"style": "info"
},
"conditionalVisibility": {
"parameterName": "resourceType",
"comparison": "isEqualTo",
"value": "microsoft.operationalinsights/workspaces"
},
"name": "aks-engine-billable-data-shown-applies-to-entire-workspace-not-just-the-cluster-info-message",
"styleSettings": {
"showBorder": true
}
},
{
"type": 11,
"content": {
"version": "LinkItem/1.0",
"style": "tabs",
"links": [
{
"id": "3b7d39f2-38c5-4586-a155-71e28e333020",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "Overview",
"subTarget": "overview",
"style": "link"
},
{
"id": "7163b764-7ab2-48b3-b417-41af3eea7ef0",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Table",
"subTarget": "table",
"style": "link"
},
{
"id": "960bbeea-357f-4c07-bdd9-6f3f123d56fd",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Namespace",
"subTarget": "namespace",
"style": "link"
},
{
"id": "7fbcc5bd-62d6-4aeb-b55e-7b1fab65716a",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Log Source",
"subTarget": "logSource",
"style": "link"
},
{
"id": "3434a63b-f3d5-4c11-9620-1e731983041c",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Diagnostic Master Node ",
"subTarget": "diagnosticMasterNode",
"style": "link"
}
]
},
"name": "tabs"
},
{
"type": 1,
"content": {
"json": "Due to querying limitation on Basic Logs, if you are using Basic Logs, this workbook provides partial data on your container log usage.",
"style": "info"
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "overview"
},
"name": "basic-logs-info-text"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"title": "Container Insights Billing Usage",
"items": [
{
"type": 1,
"content": {
"json": "<br/>\r\nThis workbook provides you a summary and source for billable data collected by [Container Insights solution.](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview)\r\n\r\nThe best way to understand how Container Insights ingests data in Log Analytics workspace is from this [article.](https://medium.com/microsoftazure/azure-monitor-for-containers-optimizing-data-collection-settings-for-cost-ce6f848aca32)\r\n\r\nIn this workbook you can:\r\n\r\n* View billable data ingested by **solution**.\r\n* View billable data ingested by **Container logs (application logs)**\r\n* View billable container logs data ingested segregated by **Kubernetes namespace**\r\n* View billable container logs data ingested segregated by **Cluster name**\r\n* View billable container log data ingested by **logsource entry**\r\n* View billable diagnostic data ingested by **diagnostic master node logs**\r\n\r\nYou can fine tune and control logging by turning off logging on the above mentioned vectors. [Learn how to fine-tune logging](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config)\r\n\r\nYou can control your master node's logs by updating diagnostic settings. [Learn how to update diagnostic settings](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"name": "workbooks-explanation-text"
},
{
"type": 1,
"content": {
"json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "no"
},
"name": "master-node-logs-are-not-enabled-msg"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "overview"
},
"name": "workbook-explanation"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "union withsource = SourceTable AzureDiagnostics, AzureActivity, AzureMetrics, ContainerLog, Perf, KubePodInventory, ContainerInventory, InsightsMetrics, KubeEvents, KubeServices, KubeNodeInventory, ContainerNodeInventory, KubeMonAgentEvents, ContainerServiceLog, Heartbeat, KubeHealth, ContainerImageInventory\r\n| where _IsBillable == true\r\n| project _BilledSize, TimeGenerated, SourceTable\r\n| summarize BillableDataBytes = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), SourceTable\r\n| render piechart\r\n\r\n\r\n",
"size": 3,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Data from Container Insights",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"xAxis": "TimeGenerated",
"createOtherGroup": 100,
"seriesLabelSettings": [
{
"seriesName": "LogManagement",
"label": "LogManagementSolution(GB)"
},
{
"seriesName": "ContainerInsights",
"label": "ContainerInsightsSolution(GB)"
}
],
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "50",
"showPin": true,
"name": "billable-data-from-ci"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "table"
},
"name": "by-datatable-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| summarize Total=sum(BillableDataBytes) by Namespace\r\n| render piechart\r\n",
"size": 3,
"showAnalytics": true,
"title": "Billable Container Log Data Per Namespace",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"tileSettings": {
"showBorder": false,
"titleContent": {
"columnMatch": "Namespace",
"formatter": 1
},
"leftContent": {
"columnMatch": "Total",
"formatter": 12,
"formatOptions": {
"palette": "auto"
},
"numberFormat": {
"unit": 17,
"options": {
"maximumSignificantDigits": 3,
"maximumFractionDigits": 2
}
}
}
},
"graphSettings": {
"type": 0,
"topContent": {
"columnMatch": "Namespace",
"formatter": 1
},
"centerContent": {
"columnMatch": "Total",
"formatter": 1,
"numberFormat": {
"unit": 17,
"options": {
"maximumSignificantDigits": 3,
"maximumFractionDigits": 2
}
}
}
},
"chartSettings": {
"createOtherGroup": 100,
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"name": "billable-data-per-namespace"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "namespace"
},
"name": "by-namespace-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 9,
"content": {
"version": "KqlParameterItem/1.0",
"crossComponentResources": [
"{resource}"
],
"parameters": [
{
"id": "d5701caa-0486-4e6f-adad-6fa5b8496a7d",
"version": "KqlParameterItem/1.0",
"name": "namespace",
"label": "Namespace",
"type": 2,
"description": "Filter data by namespace",
"isRequired": true,
"query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| project ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| distinct Namespace\r\n| project value = Namespace, label = Namespace, selected = false\r\n| sort by label asc",
"crossComponentResources": [
"{resource}"
],
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1",
"queryType": 0,
"resourceType": "{resourceType}"
}
],
"style": "pills",
"queryType": 0,
"resourceType": "{resourceType}"
},
"name": "namespace-pill"
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| join hint.strategy=shuffle (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by LogEntrySource, ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n)\r\n| extend sourceNamespace = strcat(LogEntrySource, \"/\", Namespace)\r\n| summarize Total=sum(BillableDataBytes) by sourceNamespace\r\n| render piechart",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Data Per Log-Source Type for Namespace Selected",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"createOtherGroup": 100,
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "35",
"showPin": true,
"name": "billable-data-per-log-source",
"styleSettings": {
"showBorder": true
}
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n| join hint.strategy=shuffle ( \r\nContainerLog\r\n| project _BilledSize, ContainerID, TimeGenerated, LogEntrySource\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n)\r\n| summarize ContainerLogData = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), LogEntrySource\r\n| render linechart\r\n\r\n",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Container Log Data for Namespace Selected",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"xSettings": {},
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "65",
"conditionalVisibility": {
"parameterName": "namespace:label",
"comparison": "isNotEqualTo",
"value": "<unset>"
},
"showPin": true,
"name": "billable-container-log-data-for-namespace-selected",
"styleSettings": {
"showBorder": true
}
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "logSource"
},
"name": "by-log-source-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 1,
"content": {
"json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "no"
},
"name": "master-node-logs-are-not-enabled-msg-under-masternode-tab"
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "AzureDiagnostics\r\n| summarize Total=sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), Category\r\n| render barchart\r\n\r\n\r\n",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Diagnostic Master Node Log Data",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"ySettings": {
"unit": 2,
"min": null,
"max": null
}
}
},
"customWidth": "75",
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "yes"
},
"showPin": true,
"name": "billable-data-from-diagnostic-master-node-logs"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "diagnosticMasterNode"
},
"name": "by-diagnostic-master-node-tab"
}
],
"fallbackResourceIds": [
"/subscriptions/357e7522-d1ad-433c-9545-7d8b8d72d61a/resourceGroups/ArcResources/providers/Microsoft.Kubernetes/connectedClusters/sv5-k8sdemo"
],
"fromTemplateId": "community-Workbooks/AKS/Billing Usage Divided",
"$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json"
}
Deploying 100+ node MicroK8s Kubernetes Cluster connected to Azure with Arc
Using MAAS to deploy 121 physical nodes with ubuntu installing MicroK8s from the cloud-init and connecting the cluster to Azure using Azure Arc
How long would it take to deploy 100+ physical servers with Ubuntu, create a microK8s cluster and connect it to Azure using Azure Arc?
This post walks through the process of creating and connecting a MicroK8s cluster to Azure using Azure Arc, MAAS, Ubuntu, and Micok8s.
First need a primary server to start the K8s cluster so let’s deploy one
Lets create a Tag ‘AzureConnectedMicroK8s’ so we can keep track of the servers we are deploying
While we are waiting for the deployment let’s head over to MicroK8s - Zero-ops Kubernetes for developers, edge and IoT website, and let’s look up the commands for creating a MicroK8s cluster using snaps and we want to use a specific version 1.24
sudo apt-get update
sudo apt-get upgrade -y
sudo snap install microk8s --classic --channel=1.24
sudo microk8s status --wait-ready
We need to extend the life of the join token so we reuse it for multiple servers MicroK8s - Command reference shows we do this with —token-ttl switch
sudo microk8s add-node --token-ttl 9999999I want to use the cloud-init commands with MAAS to deploy the Ubuntu and join it automatically to the microK8s cluster, so lets build that script. We will try two one the pull updates and one that doesn’t. Let’s Tag these 2 new nodes we are adding first.
#!/bin/bash
sudo apt-get update
sudo apt-get upgrade -y
sudo snap install microk8s --classic --channel=1.24
# updated content to include a random delay for adding lots of nodes
randomNumber=$((10 + $RANDOM % 240))
sleep randomNumber
sudo microk8s join 172.30.9.31:25000/8a4fb96dd8c711aaa895ba0da2e0dd91/fb637a6f6f64We can check the deployment and look for the additional nodes using kubectl command
# let’s create a Kube config file
mkdir ~/.kube
sudo microk8s kubectl config view --raw > $HOME/.kube/config
sudo usermod -aG microk8s $USER
# fix permissions for Helm warning
chmod go-r ~/.kube/config
sudo snap install kubectl --classic
kubectl version --client
kubectl cluster-info
# Add auto completion and Alias for kubectl
sudo apt-get install -y bash-completion
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
# need to restart session to use for alias
kubectl get nodes
# using k Alias
k get nodes
k get nodes -o wide
lets get K8s dashboard running
sudo microk8s enable dashboard ingress storage
# add MAAS DNS servers
sudo microk8s enable dns:172.30.0.30,172.30.0.32
# DNS trouble shooting
# https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
k -n kube-system edit service kubernetes-dashboard
# change change type from ClusterPort to NodePort ~approx line 35
# save this token for later use
k create token -n kube-system default --duration=8544h
k describe service/kubernetes-dashboard -n kube-system
Next, let’s deploy some workers from MAAS and tag the servers accordingly
#!/bin/bash
sudo apt-get update
sudo apt-get upgrade -y
sudo snap install microk8s --classic --channel=1.24
sudo microk8s join 172.30.9.31:25000/8a4fb96dd8c711aaa895ba0da2e0dd91/fb637a6f6f64 --worker
While waiting for there workers I am going to use choco to install octant. “choco install octant”. Create a copy of the linux kube-config. Creating a local ‘config’ file on my windows desktop in the user\.kube directory and then running octant.exe we can then access the dashboard via http://127.0.0.1:7777 which shows us the new workers we have just added.
Let’s work to connect the cluster to Azure using the following guide Quickstart: Connect an existing Kubernetes cluster to Azure Arc - Azure Arc | Microsoft Docs. First lets make sure the resource providers are registered
# Using Powershell
Get-AzResourceProvider -ProviderNamespace Microsoft.Kubernetes
Get-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration
Get-AzResourceProvider -ProviderNamespace Microsoft.ExtendedLocation
Adding a role assignment to my user account for ‘Kubernetes Cluster - Azure Arc Onboarding’ and ‘Azure Arc Kubernetes Cluster Admin’. Depending on your onboarding account and process this may vary.
# run on master node
sudo microk8s enable helm3
sudo snap install helm --classic
# install Azure CLI
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
az login
# User browser devicelogin code
az account show
# use 'az account set --subscription subname' to ensure you have correct subscription
az extension add --name connectedk8s
az extension add --name k8s-configuration
# using an existing resource group
az connectedk8s connect --name sv5-microk8s --resource-group ArcResources
# trouble shooting
# https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/troubleshooting#enable-custom-locations-using-service-principal
While this is deploying you can open another connection and run
“k -n azure-arc get pods,deployments”
if Azure Arc is experiencing any errors trying to create the deployments you can query the pod logs using “k -n azure-arc logs <pod-name>”
Checking Azure Resource Group we can see the resource being created and can see the total node count of 7, and the total core count of 160.
using the token we saved you can now sign into the on-premise cluster from Azure
You can now explore the namespaces, workloads, services, etc from the Azure Portal
Lets scale this out and show you how easily you can create a large-scale K8s cluster using your old hardware managed by MAAS and connected to Azure using Azure Arc. Using the same work join command as before in the cloud-init script lets deploy 10 new physical servers from MAAS and add them directly to the K8s cluster as workers.
And we can see the node count has increased to 17.
I do expect some failures here with servers as we are still onboarding some hardware and cleaning up issues but let’s grab all the ready servers and see how well Ubuntu, MAAS, MicroK8s, and Azure Arc can handle this scale out.
There Obviously adding that many servers can stress the system. Some errors around pulling the images for the networking, but just leaving them for a few hours they self-corrected and completed the deployment, and joined the cluster successfully.
We do have some internal MTU sizing issues in the switches so not all hosts were added. Looking at the Daemon sets under workloads we can see the calico and nginx pods 121 ready out of 121 deployed.
Looking at the node count we have 121 nodes added and 5452 CPU cores ready to work. That’s a lot of power you can now leverage to run container-based deployments.
Overall this was fairly easy to reuse a wide range of new and old hardware and create a large Kubernetes cluster and connect it to the Azure Portal via Azure Arc. This cluster could be used for a wide range of services including custom container hosting or Azure delivered services like Azure SQL or Machine learning.
We will continue to explore Azure Arc-connected services.
Explore
Quickstart: Connect an existing Kubernetes cluster to Azure Arc - Azure Arc | Microsoft Docs
MAAS (Metal-as-a-Service) Full HA Installation — Crying Cloud
Configure Kubernetes cluster - Azure Machine Learning | Microsoft Docs
Supporting Content
Azure Stack Hub: When you can't view/add permissions in a tenant subscription from the portal
I have noticed on a few occasions that for a tenant subscription hosted in an Azure Stack region that I am either unable to view the IAM permissions, or add a user/service principal/group for the subscription. I have needed to do this to assign a service principal as a contributor to a subscription so that I can deploy a Kubernetes cluster via AKSE.
Typically for viewing the permissions, clicking on the Refresh button does the trick. More problematic is adding permissions via the portal. Doing so renders a screen like below:
As highlighted, the animated ‘dots’ show the blade constantly trying to retrieve the data but can’t. It is actually a known issue and is highlighted in the release notes.
The remediation it offers is to use PowerShell to verify the permissions, and gives a link to the Get-AzureRmRoleAssignment CmdLet. Not helpful if you want to set permissions, so here’s a step-by-step description of what you’ll need to do.
Pre-Reqs:
Your user account is the owner of the subscriptions when they were set up.
You have the correct PowerShell for Azure Stack Hub installed
For the example shown, I am using Azure AD identities, I have more than one tenant subscription assigned to my user account and I am adding a service principal. At the end of the post, I will show the commands for adding a group or a user .
Step-By-Step
From PowerShell, connect to the tenant subscription you want to change the permissions on. The documentation is for connecting as an operator, so here’s how to do it as a user:
$Region = '<yourRegion>'
$FQDN = '<yourFQDN>'
$AADTenantName = '<yourAADTenantName>'
$envName = "AzureStack$region"
# Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name $envName -ArmEndpoint "https://management.$Region.$FQDN" `
-AzureKeyVaultDnsSuffix vault.$Region.$FQDN `
-AzureKeyVaultServiceEndpointResourceId https://vault.$Region.$FQDN
# Set your tenant name.
$AuthEndpoint = (Get-AzureRmEnvironment -Name $envName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]
# After signing in to your environment, Azure Stack cmdlets
# can be easily targeted at your Azure Stack instance.
Add-AzureRmAccount -EnvironmentName $envName -TenantId $TenantIdAs I have more than one subscription, let’s get a list so we can select the correct one:
Get-AzureRmSubscription | ft NameI got the following output:
I want to modify the AKSTest subscription:
#Set the context
$ctx = set-azurermcontext -Subscription AKSTest
$ctxRunning that command, I get :
Cool; I’ve got the correct subscription now, so let’s list the permissions assigned to it:
Get-AzureRmRoleAssignmentOK, so those are the standard permissions assigned when the subscription is created. Let’s add a service principal.
First, let’s get the service principal object
Get-AzureRmADServicePrincipal | ? {$_.DisplayName -like '*something*'} |ft DisplayName
I want to assign AKSEngineDemo
$spn = Get-AzureRmADServicePrincipal -SearchString AKSEngineDemoLets assign the service principal to the tenant subscription:
New-AzureRmRoleAssignment -ObjectId $spn.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"When I take a look in the portal now (after doing a refresh :) ), I see the following:
Here’s the commands for adding an Azure AD User:
#Find the User object you want to add
Get-AzureRmADUser | ? {$_.DisplayName -like '*something*'} |ft DisplayName
#Assign the object using the displayname of the user
$ADUser = Get-AzureRmADUser -SearchString <UserName>
New-AzureRmRoleAssignment -ObjectId $ADUser.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"Finally, how to add an Azure AD Group:
#Find the Group object you want to add
Get-AzureRmADGroup | ? {$_.DisplayName -like '*something*'} |ft DisplayName
#Assign the object using the displayname of the user
$ADGroup = Get-AzureRmADGroup -SearchString <GroupName>
New-AzureRmRoleAssignment -ObjectId $ADGroup.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"
Simplifying Kubernetes deployments on ADFS Azure Stack systems
he public preview template for Kubernetes on Azure Stack has been out for a few months now, but the ability/guidance has only been available for a short while to deploy on systems using ADFS as the identity provider. That guidance is here: https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-solution-template-kubernetes-adfs
Feel free to follow the instructions provided, as they do work, but they are fiddly.
Before you start, you have to ensure the following pre-reqs are met before running the template (taken from the doc, but with further comments from me) :
The public preview template for Kubernetes on Azure Stack has been out for a few months now, but the ability/guidance has only been available for a short while to deploy on systems using ADFS as the identity provider. That guidance is here: https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-solution-template-kubernetes-adfs
Feel free to follow the instructions provided, as they do work, but they are fiddly.
Before you start, you have to ensure the following pre-reqs are met before running the template (taken from the doc, but with further comments from me) :
Generate a SSH public and private key pair for the Linux VM’s the template creates. I use PuTTyGen. (Instructions here on generating a key: https://www.ssh.com/ssh/putty/windows/puttygen )
Have a valid tenant subscription where you are at least a contributor. The subscription/region should have enough public IP addresses (at least 2 for core deployment, you’ll need more for services you run on K8s)
Your tenant subscription will need the Key Vault service assigned within the the plan/offer
You’ll need the Kubernetes Cluster marketplace item, or you can just use the ARM template from here: https://github.com/msazurestackworkloads/azurestack-gallery/blob/master/kubernetes/template/DeploymentTemplates/azuredeploy.json
The next part of the doc talks about creating a service principal. This has to be done by an Azure Stack Operator. Currently, the Kubernetes template only supports service principals with certificates for ADFS systems, despite the fact that client secrets was introduced in 1811. Hopefully this will be addressed and supported in a future version, as it will remove the requirement for the certificate and KeyVault.
Once you’ve got the certificate, it needs to be uploaded to a KeyVault within your tenant subscription. The script provided in the doc does this for you, but you need to plug in quite a bit of information and there is the prospect of getting it wrong.
I’ve simplified the process of creating the cert, service principal, creating a key vault and uploading the cert as a secret by producing a script to do the hard work for you. To run it, you need to be an Azure Stack Operator (ability to connect to the ERCS) as well as having access to tenant subscription on the stamp of which you are a contributor.
The script does the following:
Checks if a KeyVault exists on the Azure Stack region with the name you have specified (if it does, it quits)
Creates a self-signed cert on the system you’re running the script on
Connects to the ERCS and creates a service principal using the cert
Exports the cert to a PFX file, with password of your choosing
Connects to Tenant subscription (If you have more than one subscription within the region, it will let you choose which one to deploy to)
Creates a Resource Group, a KeyVault within it and sets access policy to your user account
Uploads the certificate to the KeyVault as a secret
Dumps all the information you need for the template to screen and file
Param (
$ERCS = (Read-Host -Prompt "ERCS"),
$OutputPath = $ENV:Temp,
$SubscriptionID,
[ValidateNotNullOrEmpty()]
$appNamePrefix = "appSPN",
[ValidateNotNullOrEmpty()]
$ResourceGroup = "K8sDemoAdFsRG",
[ValidateNotNullOrEmpty()]
$KeyvaultName = "K8sDemoAdFsKV51",
[ValidateNotNullOrEmpty()]
$keyVaultSecretName = "K8sSecret",
[ValidateNotNull()]
[System.Management.Automation.PSCredential]
[System.Management.Automation.Credential()]
$ErcsCredential = (Get-Credential -Message "Enter CloudAdmin Credentials for ERCS"),
[ValidateNotNull()]
[System.Management.Automation.PSCredential]
[System.Management.Automation.Credential()]
$cloudCredential = (Get-Credential -Message "Enter Azure Stack Tenant Credentials"),
[ValidateNotNullOrEmpty()]
[Security.SecureString]$PfxPassword=(Read-Host "Enter PFX Password" -AsSecureString)
)
[bool]$GeneratePFX = $true
function write-log($logentry){
Write-output "$logentry" | out-file $detailFile -Append
Write-output "$logentry"
}
# Location to write PFX file and log file
$OutputPath = $OutputPath.Trim('\')
if (!(Test-Path $OutputPath -pathType container)) {
New-Item $OutputPath -type directory -Force
}
# Creating a PSSession to the ERCS PrivilegedEndpoint
$session = New-PSSession -ComputerName $ERCS -ConfigurationName PrivilegedEndpoint -Credential $ErcsCredential
$AzureStackInfo = Invoke-Command -Session $session -ScriptBlock { get-azurestackstampinformation }
# For Azure Stack development kit, this value is set to https://management.local.azurestack.external. We will read this from the AzureStackStampInformation output of the ERCS VM.
$ArmEndpoint = $AzureStackInfo.TenantExternalEndpoints.TenantResourceManager
# For Azure Stack development kit, this value is set to https://graph.local.azurestack.external/. We will read this from the AzureStackStampInformation output of the ERCS VM.
$GraphAudience = "https://graph." + $AzureStackInfo.ExternalDomainFQDN + "/"
# TenantID for the stamp. We will read this from the AzureStackStampInformation output of the ERCS VM.
$TenantID = $AzureStackInfo.AADTenantID
# Register an AzureRM environment that targets your Azure Stack instance
Add-AzureRMEnvironment ` -Name "azurestacktenant" ` -ArmEndpoint $ArmEndpoint
$location = $AzureStackInfo.RegionName
# Set the GraphEndpointResourceId value
$AzsEnv = Set-AzureRmEnvironment ` -Name "azurestacktenant" -GraphAudience $GraphAudience -EnableAdfsAuthentication:$true
$KeyVaultSuffix = $azsEnv.AzureKeyVaultDnsSuffix
$KeyvaultDnsName = "https://" + $KeyvaultName + "." + $KeyVaultSuffix
$KVSuffix = '/secrets/Secret1?api-version=2016-10-01'
$KVCheckURI = $KeyvaultDnsName + $KVSuffix
# This block of code in untidy, but tests whether the KeyVault namespace exists on the Stamp already (401) or not (404)
try {
(Invoke-WebRequest -Uri $KVCheckURI -ErrorAction Stop).BaseResponse
} catch [System.Net.WebException] {
# Messy, but we're not using a token to authenticate, just seeing if the name is already in use
$Status = $_.Exception.Response.StatusCode.value__
If ($Status -eq 404) {
$stat = "does not exist"
}
else
{
$stat = "exists already"
}
Write-Debug ("KeyVault Namespace {0} {1} in Region {2}" -f $KeyvaultDnsName, $stat, $Location)
}
# Only carry on if the KeyVault namespace doesn't exist on the Stamp
If ($Status -eq 404) {
Write-Debug "Creating Self-signed cert and new Graph APplication..."
# This produces a self signed cert for testing purposes. It is preferred to use a managed certificate for this.
if ($GeneratePFX) {
$cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" -Subject "CN=$appNamePrefix" -KeySpec KeyExchange
$ServicePrincipal = Invoke-Command -Session $session {New-GraphApplication -Name $args[0] -ClientCertificates $args[1]} -ArgumentList $appNamePrefix,$cert
}
else {
$ServicePrincipal = Invoke-Command -Session $session {New-GraphApplication -Name $args[0] -GenerateClientSecret} -ArgumentList $appNamePrefix
}
$session|remove-pssession
$SPNName = $ServicePrincipal.ApplicationName
$PFXFile = "$OutputPath\$SPNName.pfx"
$detailfile = "$OutputPath\$SPNName-details.txt"
write-Log "Client Id : $($ServicePrincipal.ClientId)"
if ($GeneratePFX) { write-output "Cert Thumbprint : $($ServicePrincipal.Thumbprint)"}
else { write-output "Client Secret : $($ServicePrincipal.ClientSecret)"}
write-Log "Application Name : $($ServicePrincipal.ApplicationName)"
write-Log "TenantID : $TenantID"
write-Log "ARM EndPoint : $ArmEndpoint"
write-Log "Admin Endpoint : $AdminEndpoint"
write-Log "Graph Audience : $GraphAudience"
# Now Export the cert to PFX
if ($GeneratePFX){
# enter a password for the PFX file...
$pw = $PfxPassword
# Store the cert in the designated output directory
Export-PfxCertificate -cert $cert -FilePath $PFXFile -Password $pw
write-Log "PFX Certificate : $PFXFile"
}
# Connect to the Stamp
If ($SubscriptionID) {
$AzsUser = Login-AzureRmAccount -Environment azurestacktenant -Credential $Cloudcreds -Subscription $subscriptionId
}
else
{
$AzsUser = Login-AzureRmAccount -Environment azurestacktenant -Credential $Cloudcreds
$Subs = Get-AzureRmSubscription
# Show a list of subs if more than one is available
If ($Subs.Count -gt 1) {
$context = $Subs | Out-GridView -PassThru
Set-AzureRmContext -Subscription $context
}
}
#Get the SID for the user account you've used to connect to the Subscription
$adfsuserID = $null
try {
# using the get-azurermaduser means the script can be used on non-domain joined systems :)
$adfsuserID = (get-azurermaduser -UserPrincipalName $azsuser.Context.Account.Id).AdfsID
}
catch {
}
# This can be used for currently logged in user:
<#
if (-not $adfsuserID) {
$Filter = "name = '" + $env:USERNAME + "' AND domain = '" + $env:USERDOMAIN + "'"
$adfsuserID = (Get-WmiObject win32_useraccount -Filter "$Filter").SID
}
#>
# Create new Resource group and key vault
New-AzureRmResourceGroup -Name $ResourceGroup -Location $location -Force
New-AzureRmKeyVault -VaultName $KeyvaultName -ResourceGroupName $ResourceGroup -Location $location -EnabledForTemplateDeployment
Set-AzureRmKeyVaultAccessPolicy -VaultName $KeyvaultName -ResourceGroupName $ResourceGroup -ObjectId $adfsuserID -BypassObjectIdValidation -PermissionsToKeys all -PermissionsToSecrets all
#Convert the secure pw to something that can be used
$BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)
$password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)
$certContentInBytes = [io.file]::ReadAllBytes($PFXFile)
$pfxAsBase64EncodedString = [System.Convert]::ToBase64String($certContentInBytes)
$jsonObject = @"
{
"data": "$pfxAsBase64EncodedString",
"dataType" :"pfx",
"password": "$password"
}
"@
$jsonObjectBytes = [System.Text.Encoding]::UTF8.GetBytes($jsonObject)
$jsonEncoded = [System.Convert]::ToBase64String($jsonObjectBytes)
$secret = ConvertTo-SecureString -String $jsonEncoded -AsPlainText -Force
$keyVaultSecret = Set-AzureKeyVaultSecret -VaultName $KeyvaultName -Name $keyVaultSecretName -SecretValue $secret
#Give the new Service Principal Contributor rights to the Subscription
New-AzureRmRoleAssignment -ApplicationID ($ServicePrincipal.ClientId) -RoleDefinitionName "Contributor" -Scope "/subscriptions/$($context.Id)"
Write-Log ('')
Write-Log "Service principal clientId : $($ServicePrincipal.ClientId)"
Write-Log "Key vault resource group : $ResourceGroup "
Write-Log "Key vault name : $KeyvaultName"
Write-Log "Key vault secret : $keyVaultSecretName"
$detailfile
}
else {
write-Error "Certificate and Keyvault processing halted as KeyVault namespace already exists in this region. Please try another name"
}
When you run the script, you should hopefully see output resembling this:
I’ve formatted it so that you can copy paste it into the template. I could have created a parameter file, but for my purposes this was fine.
For a deeper understanding of whats happening when deploying the template, take a look at Ned Bellavance’s great post here: https://nedinthecloud.com/2019/02/19/azure-stack-kubernetes-cluster-is-not-aks/ .
Topic Search
-
Securing TLS in WAC (Windows Admin Center) https://t.co/klDc7J7R4G
Posts by Date
- March 2025 1
- February 2025 1
- October 2024 1
- August 2024 1
- July 2024 1
- October 2023 1
- September 2023 1
- August 2023 3
- July 2023 1
- June 2023 2
- May 2023 1
- February 2023 3
- January 2023 1
- December 2022 1
- November 2022 3
- October 2022 7
- September 2022 2
- August 2022 4
- July 2022 1
- February 2022 2
- January 2022 1
- October 2021 1
- June 2021 2
- February 2021 1
- December 2020 2
- November 2020 2
- October 2020 1
- September 2020 1
- August 2020 1
- June 2020 1
- May 2020 2
- March 2020 1
- January 2020 2
- December 2019 2
- November 2019 1
- October 2019 7
- June 2019 2
- March 2019 2
- February 2019 1
- December 2018 3
- November 2018 1
- October 2018 4
- September 2018 6
- August 2018 1
- June 2018 1
- April 2018 2
- March 2018 1
- February 2018 3
- January 2018 2
- August 2017 5
- June 2017 2
- May 2017 3
- March 2017 4
- February 2017 4
- December 2016 1
- November 2016 3
- October 2016 3
- September 2016 5
- August 2016 11
- July 2016 13