Skip to content

Commit

Permalink
POL-1433 Use Flexera CCO Costs for Azure Rightsize Managed Disks (#2892)
Browse files Browse the repository at this point in the history 
* update

* fix

* update

* fix

* update

* fix

* update

* update

* update

* update
  • Loading branch information
@XOmniverse
XOmniverse authored Jan 7, 2025
1 parent 02d6067 commit f6f87c1
Show file tree
Hide file tree
Showing 6 changed files with 212 additions and 67 deletions.
5 changes: 5 additions & 0 deletions cost/azure/rightsize_managed_disks/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## v2.5.0

- Savings are now calculated using cost data stored in Flexera Cloud Cost Optimization instead of only via Azure list price.
- Currency conversion functionality has been removed. It is no longer needed due to actual cost data stored in Flexera Cloud Cost Optimization being used to assess cost and savings.

## v2.4.2

- Added `hide_skip_approvals` field to the info section. It dynamically controls "Skip Action Approvals" visibility.
Expand Down
8 changes: 6 additions & 2 deletions cost/azure/rightsize_managed_disks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,13 @@ Note: This policy template does not currently produce recommendations or reporti

### Policy Saving Details

The policy includes the estimated monthly savings. The estimated monthly savings are recognized if the resource is resized to the suggested size.
The policy includes the estimated monthly savings. The estimated monthly savings are recognized if the resource is resized to the suggested size. The `Estimated Monthly Savings` is calculated via the following:

- The `Estimated Monthly Savings` is calculated by obtaining the price of the disk per month from the Azure Pricing API.
- The `monthly list price` of the current disk type obtained via the Azure Pricing API.
- The `real monthly cost of the disk` is calculated by multiplying the amortized cost of the disk for 1 day, as found within Flexera CCO, by 30.44, which is the average number of days in a month.
- The percentage difference between the two is calculated by dividing the `real monthly cost of the disk` by the `monthly list price` of the current disk type.
- The `monthly list price of the new disk type` is multiplied by the above percentage to get an `estimated real monthly cost of the new disk` type under the assumption that any discounts or other changes from list price that applied to the old disk type will also apply to the new one.
- The savings is then calculated by subtracting the `estimated real monthly cost of the new disk type` from the `real monthly cost of the disk`.
- The incident message detail includes the sum of each resource `Estimated Monthly Savings` as `Potential Monthly Savings`.

## Input Parameters
Expand Down
260 changes: 198 additions & 62 deletions cost/azure/rightsize_managed_disks/azure_rightsize_managed_disks.pt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ category "Cost"
severity "low"
default_frequency "monthly"
info(
version: "2.4.2",
version: "2.5.0",
provider: "Azure",
service: "Managed Disks",
policy_set: "Rightsize Storage",
Expand Down Expand Up @@ -265,6 +265,44 @@ datasource "ds_applied_policy" do
end
end

datasource "ds_billing_centers" do
request do
auth $auth_flexera
host rs_optima_host
path join(["/analytics/orgs/", rs_org_id, "/billing_centers"])
query "view", "allocation_table"
header "Api-Version", "1.0"
header "User-Agent", "RS Policies"
ignore_status [403]
end
result do
encoding "json"
collect jmes_path(response, "[*]") do
field "href", jmes_path(col_item, "href")
field "id", jmes_path(col_item, "id")
field "name", jmes_path(col_item, "name")
field "parent_id", jmes_path(col_item, "parent_id")
end
end
end

# Gather top level billing center IDs for when we pull cost data
datasource "ds_top_level_bcs" do
run_script $js_top_level_bcs, $ds_billing_centers
end

script "js_top_level_bcs", type: "javascript" do
parameters "ds_billing_centers"
result "result"
code <<-EOS
filtered_bcs = _.filter(ds_billing_centers, function(bc) {
return bc['parent_id'] == null || bc['parent_id'] == undefined
})
result = _.compact(_.pluck(filtered_bcs, 'id'))
EOS
end

datasource "ds_min_used_disk_space_pct" do
run_script $js_min_used_disk_space_pct
end
Expand Down Expand Up @@ -883,7 +921,6 @@ datasource "ds_azure_md_pricing" do
end
end

# Gather local currency info
datasource "ds_currency_reference" do
request do
host "raw.githubusercontent.com"
Expand All @@ -908,95 +945,183 @@ datasource "ds_currency_code" do
end
end

datasource "ds_currency_target" do
run_script $js_currency_target, $ds_currency_reference, $ds_currency_code
datasource "ds_currency" do
run_script $js_currency, $ds_currency_reference, $ds_currency_code
end

script "js_currency_target", type:"javascript" do
script "js_currency", type:"javascript" do
parameters "ds_currency_reference", "ds_currency_code"
result "result"
code <<-EOS
// Default to USD if currency is not found
result = ds_currency_reference['USD']
symbol = "$"
separator = ","
if (ds_currency_code['value'] != undefined) {
if (ds_currency_reference[ds_currency_code['value']] != undefined) {
symbol = ds_currency_reference[ds_currency_code['value']]['symbol']
if (ds_currency_reference[ds_currency_code['value']]['t_separator'] != undefined) {
separator = ds_currency_reference[ds_currency_code['value']]['t_separator']
} else {
separator = ""
}
}
}
if (ds_currency_code['value'] != undefined && ds_currency_reference[ds_currency_code['value']] != undefined) {
result = ds_currency_reference[ds_currency_code['value']]
result = {
symbol: symbol,
separator: separator
}
EOS
end

# Branching logic:
# This datasource returns an empty array if the target currency is USD.
# This prevents ds_currency_conversion from running if it's not needed.
datasource "ds_conditional_currency_conversion" do
run_script $js_conditional_currency_conversion, $ds_currency_target
datasource "ds_azure_disk_subscriptions" do
run_script $js_azure_disk_subscriptions, $ds_azure_disks_with_metrics_thresholds_filtered
end

script "js_conditional_currency_conversion", type: "javascript" do
parameters "ds_currency_target"
script "js_azure_disk_subscriptions", type: "javascript" do
parameters "ds_azure_disks_with_metrics_thresholds_filtered"
result "result"
code <<-EOS
result = []
// Make the request only if the target currency is not USD
if (ds_currency_target['code'] != 'USD') {
result = [1]
}
result = _.compact(_.uniq(_.pluck(ds_azure_disks_with_metrics_thresholds_filtered, 'subscriptionID')))
EOS
end

datasource "ds_currency_conversion" do
# Only make a request if the target currency is not USD
iterate $ds_conditional_currency_conversion
datasource "ds_azure_disk_costs" do
iterate $ds_azure_disk_subscriptions
request do
host "api.xe-auth.flexeraeng.com"
path "/prod/{proxy+}"
query "from", "USD"
query "to", val($ds_currency_target, 'code')
query "amount", "1"
# Ignore currency conversion if API has issues
ignore_status [400, 404, 502]
run_script $js_azure_disk_costs, iter_item, $ds_top_level_bcs, rs_org_id, rs_optima_host
end
result do
encoding "json"
field "from", jmes_path(response, "from")
field "to", jmes_path(response, "to")
field "amount", jmes_path(response, "amount")
field "year", jmes_path(response, "year")
collect jmes_path(response, "rows[*]") do
field "resourceID", jmes_path(col_item, "dimensions.resource_id")
field "cost", jmes_path(col_item, "metrics.cost_amortized_unblended_adj")
end
end
end

datasource "ds_currency" do
run_script $js_currency, $ds_currency_target, $ds_currency_conversion
script "js_azure_disk_costs", type:"javascript" do
parameters "subscription_id", "ds_top_level_bcs", "rs_org_id", "rs_optima_host"
result "request"
code <<-EOS
//Get Start and End dates
start_date = new Date()
start_date.setDate(start_date.getDate() - 3)
start_date = start_date.toISOString().split("T")[0]
end_date = new Date()
end_date.setDate(end_date.getDate() - 2)
end_date = end_date.toISOString().split("T")[0]
var request = {
auth: "auth_flexera",
host: rs_optima_host,
verb: "POST",
path: "/bill-analysis/orgs/" + rs_org_id + "/costs/select",
body_fields: {
"dimensions": [ "resource_id" ],
"granularity": "day",
"start_at": start_date,
"end_at": end_date,
"metrics": [ "cost_amortized_unblended_adj" ],
"billing_center_ids": ds_top_level_bcs,
"limit": 100000,
"filter": {
"type": "and",
"expressions": [
{
"type": "or",
"expressions": [
{
"dimension": "service",
"type": "equal",
"value": "Microsoft.Compute"
},
{
"dimension": "service",
"type": "equal",
"value": "microsoft.compute"
}
]
},
{
"type": "or",
"expressions": [
{
"dimension": "meter_category",
"type": "equal",
"value": "Storage"
},
{
"dimension": "meter_category",
"type": "equal",
"value": "storage"
}
]
},
{
"type": "or",
"expressions": [
{
"dimension": "resource_id",
"type": "substring",
"substring": "/providers/Microsoft.Compute/disks/"
},
{
"dimension": "resource_id",
"type": "substring",
"substring": "/providers/microsoft.compute/disks/"
}
]
},
{
"dimension": "vendor_account",
"type": "equal",
"value": subscription_id
}
]
}
},
headers: {
"User-Agent": "RS Policies",
"Api-Version": "1.0"
},
ignore_status: [400]
}
EOS
end

script "js_currency", type:"javascript" do
parameters "ds_currency_target", "ds_currency_conversion"
datasource "ds_azure_disk_costs_grouped" do
run_script $js_azure_disk_costs_grouped, $ds_azure_disk_costs
end

script "js_azure_disk_costs_grouped", type: "javascript" do
parameters "ds_azure_disk_costs"
result "result"
code <<-EOS
result = ds_currency_target
result['exchange_rate'] = 1
// Multiple a single day's cost by the average number of days in a month.
// The 0.25 is to account for leap years for extra precision.
cost_multiplier = 365.25 / 12
if (ds_currency_conversion.length > 0) {
currency_code = ds_currency_target['code']
current_month = parseInt(new Date().toISOString().split('-')[1])
// Group cost data by resourceId for later use
result = {}
conversion_block = _.find(ds_currency_conversion[0]['to'][currency_code], function(item) {
return item['month'] == current_month
})
_.each(ds_azure_disk_costs, function(item) {
id = item['resourceID'].toLowerCase()
if (conversion_block != undefined) {
result['exchange_rate'] = conversion_block['monthlyAverage']
}
}
if (result[id] == undefined) { result[id] = 0.0 }
result[id] += item['cost'] * cost_multiplier
})
EOS
end

datasource "ds_disk_rightsizing_recommendations" do
run_script $js_disk_rightsizing_recommendations, $ds_currency, $ds_azure_disks_with_metrics_thresholds_filtered, $ds_azure_md_tier_types, $ds_azure_md_pricing, $ds_applied_policy, $ds_min_used_disk_space_pct, $param_stats_lookback, $param_min_savings, $param_min_used_disk_iops_pct, $param_min_used_disk_throughput_pct, $param_stats_aggregation_for_disk_iops, $param_stats_aggregation_for_disk_throughput, $param_recommend_hdd_tier
run_script $js_disk_rightsizing_recommendations, $ds_currency, $ds_azure_disk_costs_grouped, $ds_azure_disks_with_metrics_thresholds_filtered, $ds_azure_md_tier_types, $ds_azure_md_pricing, $ds_applied_policy, $ds_min_used_disk_space_pct, $param_stats_lookback, $param_min_savings, $param_min_used_disk_iops_pct, $param_min_used_disk_throughput_pct, $param_stats_aggregation_for_disk_iops, $param_stats_aggregation_for_disk_throughput, $param_recommend_hdd_tier
end

script "js_disk_rightsizing_recommendations", type: "javascript" do
parameters "ds_currency", "ds_azure_disks_with_metrics_thresholds_filtered", "ds_azure_md_tier_types", "ds_azure_md_pricing", "ds_applied_policy", "ds_min_used_disk_space_pct", "param_stats_lookback", "param_min_savings", "param_min_used_disk_iops_pct", "param_min_used_disk_throughput_pct", "param_stats_aggregation_for_disk_iops", "param_stats_aggregation_for_disk_throughput", "param_recommend_hdd_tier"
parameters "ds_currency", "ds_azure_disk_costs_grouped", "ds_azure_disks_with_metrics_thresholds_filtered", "ds_azure_md_tier_types", "ds_azure_md_pricing", "ds_applied_policy", "ds_min_used_disk_space_pct", "param_stats_lookback", "param_min_savings", "param_min_used_disk_iops_pct", "param_min_used_disk_throughput_pct", "param_stats_aggregation_for_disk_iops", "param_stats_aggregation_for_disk_throughput", "param_recommend_hdd_tier"
result "result"
code <<-'EOS'
var hddDiskSizes = [32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32767]
Expand Down Expand Up @@ -1219,12 +1344,23 @@ script "js_disk_rightsizing_recommendations", type: "javascript" do
newResourceTypePriceCurrency = fittingDiskPrice.currencyCode
} else continue;
if (monthlySavings < param_min_savings) {
continue;
// Find cost from cost data and adjust values accordingly
if (ds_azure_disk_costs_grouped[disk['id'].toLowerCase()] && currentDiskPrice['pricePerUnit']) {
resourceTypeMonthlyPrice = ds_azure_disk_costs_grouped[disk['id'].toLowerCase()]
percentage = resourceTypeMonthlyPrice / currentDiskPrice['pricePerUnit']
newResourceTypeMonthlyPrice = newResourceTypeMonthlyPrice * percentage
monthlySavings = resourceTypeMonthlyPrice - newResourceTypeMonthlyPrice
} else {
resourceTypeMonthlyPrice = 0
newResourceTypeMonthlyPrice = 0
monthlySavings = 0
}
if (monthlySavings < param_min_savings) { continue }
tags = []
if(disk.tags){
if (disk.tags) {
if (typeof(disk.tags) == 'object') {
_.each(Object.keys(disk.tags), function(key) {
tags.push([key, "=", disk.tags[key]].join(''))
Expand All @@ -1244,19 +1380,19 @@ script "js_disk_rightsizing_recommendations", type: "javascript" do
resourceTypeGiB: currentDiskData.sizeGiB,
resourceTypeIOPS: currentDiskData.IOPS,
resourceTypeThroughput: currentDiskData.throughput,
resourceTypeMonthlyPrice: currentDiskPrice.pricePerUnit * ds_currency.exchange_rate,
resourceTypePriceCurrency: ds_currency.code,
resourceTypeMonthlyPrice: Math.round(resourceTypeMonthlyPrice * 1000) / 1000,
resourceTypePriceCurrency: ds_currency['symbol'],
newResourceType: newResourceType,
newResourceTypeRedundancy: newResourceTypeRedundancy,
newResourceTypeGiB: newResourceTypeGiB,
newResourceTypeIOPS: newResourceTypeIOPS,
newResourceTypeThroughput: newResourceTypeThroughput,
newResourceTypeMonthlyPrice: newResourceTypeMonthlyPrice * ds_currency.exchange_rate,
newResourceTypePriceCurrency: ds_currency.code,
newResourceTypeMonthlyPrice: Math.round(newResourceTypeMonthlyPrice * 1000) / 1000,
newResourceTypePriceCurrency: ds_currency['symbol'],
resourceKind: disk.resourceType,
region: disk.region,
monthlySavings: monthlySavings.toFixed(2) * ds_currency.exchange_rate,
savingsCurrency: ds_currency.code,
monthlySavings: Math.round(monthlySavings * 1000) / 1000,
savingsCurrency: ds_currency['symbol'],
iopsPctMaximum: Math.round(disk.iopsConsumedPctMax),
iopsPctAverage: Math.round(disk.iopsConsumedPctAvg),
iopsPctP90: Math.round(disk.iopsConsumedPctP90),
Expand Down
2 changes: 1 addition & 1 deletion cost/azure/rightsize_managed_disks/azure_rightsize_managed_disks_meta_parent.pt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ category "Meta"
default_frequency "15 minutes"
info(
provider: "Azure",
version: "2.4.2", # This version of the Meta Parent Policy Template should match the version of the Child Policy Template as it appears in the Catalog for best reliability
version: "2.5.0", # This version of the Meta Parent Policy Template should match the version of the Child Policy Template as it appears in the Catalog for best reliability
publish: "true",
deprecated: "false",
hide_skip_approvals: "true"
Expand Down
2 changes: 1 addition & 1 deletion data/policy_permissions_list/master_policy_permissions_list.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3833,7 +3833,7 @@
{
"id": "./cost/azure/rightsize_managed_disks/azure_rightsize_managed_disks.pt",
"name": "Azure Rightsize Managed Disks",
"version": "2.4.2",
"version": "2.5.0",
"providers": [
{
"name": "azure_rm",
Expand Down
Loading

0 comments on commit f6f87c1

Please sign in to comment.