hopsworks.cost #

Per-namespace or per-department cost over one slice of a window, for plotting cost over time.

An hour window is split into 10-minute slices, a day into hours, a week and a month into days, and a year into calendar months. start and end are ISO-8601 UTC timestamps. A bucket with observed false has no data, for example because OpenCost was not running yet or the data is past retention, so its cost is unknown rather than zero. A bucket with partial true has not ended yet, so its cost is still growing.

How many of the buckets of a window have any cost data.

The totals and rows of a NamespaceCosts cover only the observed buckets, so a window with observed_buckets below total_buckets reports less than the full window. first_observed is the ISO-8601 UTC start of the first observed bucket, and None when no bucket has data.

Cost of one Kubernetes namespace, or one department, over a time window, as computed by OpenCost.

Costs are in the currency of the enclosing NamespaceCosts. kind is NAMESPACE for a namespace, IDLE for node capacity no pod requested (namespace __idle__), and UNMOUNTED for unmounted volumes and load balancers (namespace __unmounted__). A NAMESPACE row carries the department_id and department_name of the projects in it, both None when the namespace is no project's or is shared by projects of different departments. The department view has DEPARTMENT rows, one per department with namespace None, and an UNASSIGNED row for the namespaces that have no single department, alongside the same IDLE and UNMOUNTED rows. namespace_count is the number of namespaces summed into a DEPARTMENT or UNASSIGNED row, and None on every other row. A department holds its projects' whole history, so a project moved to another department moves its past cost with it. other_cost sums load balancer, shared and external costs. The cost columns sum to total_cost, and costs can be slightly negative after OpenCost's adjustments. gpu_workload_cost and cpu_workload_cost split total_cost by workload instead of by resource: a workload (Job, Deployment, StatefulSet, SparkApplication and the like) that requested GPU-hours in a bucket counts its whole cost, CPU, RAM and storage included, as gpu_workload_cost, and every other workload counts as cpu_workload_cost. The two sum to total_cost for a NAMESPACE, DEPARTMENT or UNASSIGNED row and are 0 for IDLE and UNMOUNTED rows, whereas gpu_cost is the cost of the GPUs alone. Pods without a controller share one workload per namespace, so they are classed together. The *_hours_requested and *_hours_used columns are CPU core-hours and RAM byte-hours. Efficiencies are used divided by requested hours, cost-weighted over CPU and RAM for total_efficiency, and None when nothing was requested. cpu_no_requests and ram_no_requests mean the namespace used that resource without requesting any, where OpenCost would report 100% efficiency. share is the fraction of the window's total cost, and None when that total is not positive, when the row is inside a CostBucket, or in a project report from CostsApi.get_project_costs, whose only row would always be 1.

Kubernetes cost per namespace, or per department, over a time window.

The cost is computed by OpenCost and proxied by Hopsworks. Each Hopsworks project runs in its own namespace, so a namespace row is the compute and storage cost of that project. In the administrator reports, total_cost and idle_cost cover the whole cluster, and efficiency is the cost-weighted efficiency of all namespace rows. In a project report from CostsApi.get_project_costs they cover that project alone: items and the bucket rows are the project's NAMESPACE row, and idle_cost is 0. gpu_workload_cost and cpu_workload_cost are the sums of the namespace rows' workload split, so with idle_cost and the UNMOUNTED row they add up to total_cost. start and end are ISO-8601 UTC timestamps. The totals and items cover only the buckets with data, as reported by coverage, and items are the per-row sums of the bucket rows, most expensive first. The department view has the same totals as the namespace view for the same window, with its namespace rows grouped into DEPARTMENT and UNASSIGNED rows.

Use CostsApi.get_namespace_costs, CostsApi.get_department_costs or CostsApi.get_project_costs to get an instance of this class.

to_pandas() -> pd.DataFrame

import hopsworks

hopsworks.login()
costs = hopsworks.get_costs_api().get_namespace_costs("month")

df = costs.to_pandas().sort_values("total_cost", ascending=False)