Volume usage metrics support

[gnufied]

This came up during sig-storage f2f, we currently don't have a reliable way of getting volume usage metrics from CSI plugins. One can argue that - one can use "du" or "stat" from CO to get those metrics and this need not be part of CSI specs, but I believe that how "usage" metric of a volume is computed depends on volume type and should be part of CSI spec.

In Kubernetes at least currently - each volume type implements its own metric implementation.

[gnufied]

cc @jdef @saad-ali

[jdef]

FWIW there's a couple of basic measurements that I think are important for quota/usage within a volume:

1. data capacity
2. metadata capacity (read: inodes (fstype-specific) there may be a finite amount of these to go around)

I wonder if it's worth reporting raw values (MB, GB, inode counts) or percentages. I guess the benefit of implementing metrics for these things in a CSI plugin is that it pushes knowledge (and any related tooling dependencies) of the underlying storage further into the plugin layer (as opposed to the CO layer). For BLOCK volumes I can think of a single scenario in which capacity might be accurately measured by a plugin: sparse BLOCK volumes (e.g. dynamically expanding virtual disk); otherwise BLOCKs are generally opaque and it's not clear to me how a plugin would measure usage of one.

Pushing this complexity into the plugin layer simplifies the CO at the expense of plugins. However it's also arguable that storage-domain specific knowledge should be captured at the plugin layer. Is higher complexity in the plugin layer acceptable in this case?

It's probably also worth thinking about this in the context of "volume health checks", something that we (the COs, at least) discussed a while ago but was put on the back burner.

I'd like to see some use cases spelled out for why this is important. Some initial thoughts:

(a) The CO should monitor available volume capacity so that it may emit an administrative alert / event.

(b) The CO should monitor available volume capacity for the purposes of auto-expanding a volume when available capacity falls below some threshold.

Are there others?

[gnufied]

Currently in Kubernetes we are reporting following metrics:

1. volume_stats_used_bytes
2. volume_stats_available_bytes
3. volume_stats_inodes_free
4. volume_stats_inodes
5. volume_stats_inodes_used
6. volume_stats_capacity_bytes

I think at minimum we will want similar or same metrics from CSI.

The other use case we have is - quite simply user wants to monitor its PVC usage(volume) in some kind of metrics gathering system. Any action that user may take is upto the user but monitoring the volumes for usage gets them halfway there.

[jieyu]

IIUC, the reason those metrics need to be in the spec because CO needs to be aware of those metrics so that it can react on them? For the list of metrics you listed above which all seems capacity/usage related, I am wondering if we should think about how to improve GetCapacity call?

Also, I feel that we should separate those metrics that CO needs to know vs. those metrics that is SP specific and CO does not need to know. For the latter, IMO, it's SP's responsibility to expose those through some metrics endpoint of the plugin using some well known format (e.g., prometheus or statsd). For the former, I'd like to understand what the CO's reaction will be based on the metrics value. Is the intention to not issue CreateVolume call if there's no inode/capacity left? Or there're some other reasons?

[cpuguy83]

Agree with @jieyu, though it can be helpful to have a way for the plugin to surface at least some (opaque to the CO) data that can easily be presented to a user.

[saad-ali]

No major concerns from community call. GetCapacity is currently cluster wide not volume specific. Someone suggested using ListVolumes and making capacity related fields on returned volume object.

[jieyu]

Can someone add some context about why CO cares about the per volume metrics information? or this is just for display purpose?

[cpuguy83]

Display only here. Brian Goff

…

On Jun 20, 2018, at 11:35, Jie Yu ***@***.***> wrote: Can someone add some context about why CO cares about the per volume metrics information? or this is just for display purpose? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[gnufied]

Monitoring, alerting is primarly goal for us.

[cpuguy83]

@gnufied Using this for monitoring/alerting doesn't seem quite right and I agree with @jieyu that we should push that into the SP.

[gnufied]

From original proposal where it was introduced in k8s - https://github.com/kubernetes/community/pull/855/files

The background and motivation section talks about this in more detail than I have captured here.

Monitoring resource usage is critical for users/admins to manage their systems. For storage, there are three levels of metrics:

Node-level: the capacity, used, and allocatable storage on local host
Pod-level: Pod volume could be network attached (GCE PD, AWS EBS etc.) or local (emptyDir, hostPath, or local storage volume). Each volume has a total capacity and used bytes
Container-level: How much storage is used by each container’s writable layer and logs (standard input/output).
All the above metrics are used for operation of Kubernetes internal components and core utilities as explains below.

Eviction manager checks the allocatable local storage (scratch space) and take actions. Scheduler also needs the node-level local storage capacity and availability to determine how to place pods on nodes
In 1.7, an alpha feature is added to isolation emptyDir volume usage and container’s overlay in local storage. User can set a sizeLimit for Pod EmptyDir Volume and eviction manager will take action if the emptyDir volume usage exceeds the configured size. Similarly, user can set request/limit on container’s overlay and eviction manager can take actions based on the monitoring data and the limits.
There is also a proposal targeting for 1.8 volume resize proposal to dynamically resize the volume (network attached) if the disk resource is insufficient. It requires a monitoring pipeline to dynamically check the the disk usage for each volume (represented by PVC)

IMO - i do not see how these metrics can be "pushed" into SP where for example SP is local disk or NFS. In absence of such metric from CO/CSI - what users will have to do is run some kind of daemon on each node and calculate volume usage with SP specific information and then correlate that with CO specific information (such as PVC).

Also I was not entirely correct that metrics are currently being used only for display but it appears that k8s uses local ephemeral storage capacity information for pod eviction at very least.

cc @jingxu97

[sandaymin123]

@gnufied Can we also support this for Flex?

[ddebroy]

I think we should not have OS specific stats like the following

volume_stats_inodes_free
volume_stats_inodes
volume_stats_inodes_used

mentioned above.

The inode terminology will not directly apply to Windows filesystems.