Resource quotas allow you to place limits on how many resources a particular namespace can use. Depending on how you have chosen to use namespaces in your organization, they can give you a powerful way to limit the resources that are used by a particular team, application, or group of applications, while still giving developers the freedom to tweak the resource limits of each individual container.
A resource quota, defined by a ResourceQuota object, provides
constraints that limit aggregate resource consumption per namespace. It
can limit the quantity of objects that can be created in a namespace by
type, as well as the total amount of compute resources that may be
consumed by resources in that namespace.
Don’t get confused between Kubernetes Resource Quota and Resource Limit, The resource quota is applied on the namespace while the resource limit is applied on the containers.
Resource quotas for namespaces
- In Kubernetes, resource quotas are managed by an admission controller.
- Resource quotas are a useful tool when you want to control the resource costs of different teams or applications, but still want to achieve the utilization benefits of scheduling multiple workloads to the same cluster.
- If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code403 FORBIDDENwith a message explaining the constraint that would have been violated.
- This controller tracks the use of resources such as pods and services, and if a limit is exceeded, it prevents new resources from being created.
- The resource quota admission controller is configured by one or moreResourceQuotaobjects created in the namespace.
- If quota is enabled in a namespace for compute resources likecpuandmemory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation
Resource quota types
There aredifferent types of quota we can manage and control. The categories are compute, storage, and objects.
Compute resource quota
Compute resources are CPU and memory. For each one, you can specify a
limit or request a certain amount. Here is the list of compute-related
fields. Note that requests.cpu can be specified as just cpu, and
requests.memory can be specified as just memory:
limits.cpu: Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this valuelimits.memory: Across all pods in a non-terminal state, the sum of memory limits cannot exceed this valuerequests.cpu: Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this valuerequests.memory: Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value
Storage resource quota
The storage resource quota type is a little more complicated. There are two entities you can restrict per namespace: the amount of storage and the number of persistent volume claims. However, in addition to just globally setting the quota on the total storage or the total number of persistent volume claims, you can also do that per storage class. The notation for storage class resource quota is a little verbose, but it gets the job done:
requests.storage: The total amount of requested storage across all persistent volume claimspersistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace.storageclass.storage.k8s.io/requests.storage: The total amount of requested storage across all persistent volume claims associated with the storage class name.storageclass.storage.k8s.io/persistentvolumeclaims: The maximum number of persistent volume claims allowed in the namespace that are associated with the storage class name
Kubernetes 1.8 added alpha support for ephemeral storage quotas too:
requests.ephemeral-storage: The total amount of requested ephemeral storage across all pods in the namespace claimslimits.ephemeral-storage: The total amount of limits for ephemeral storage across all pods in the namespace claims
Object count quota
Kubernetes has another category of resource quotas, which is API objects. Since Kubernetes 1.9 you can restrict the number of any namespaced resource (prior to that coverage of objects that can be restricted was a little spotty). The syntax is interesting:
count/<resource>.<group>for resources from non-core groupscount/<resource>for resources from the core group
Here are some objects you may want to limit (note that deployments can be limited for two separate API groups):
count/persistentvolumeclaims
count/services
count/secrets
count/configmaps
count/replicationcontrollers
count/deployments.apps
count/replicasets.apps
count/statefulsets.apps
count/jobs.batch
count/cronjobs.batch
It is also possible to do generic object count quota on a limited set of
resources such as pods, services, secrets, configmaps etc. For example,
pods quota counts and enforces a maximum on the number of pods
created in a single namespace that are not terminal. You might want to
set a pods quota on a namespace to avoid the case where a user creates
many small pods and exhausts the cluster’s supply of Pod IPs
Example: Define CPU quota for a namespace
As quotas will affect all the pods within a namespace, we will start by
creating a new namespace usingkubectl:
[root@controller ~]# kubectl create namespace quota-example
namespace/quota-example created
Next we will create a YAML file with a CPU quota limit and assign this to the newly created namespace, in this example we have only defined quota limit for CPU for 1 core but you can also add limit for other resource types such as memory, pod count etc.
[root@controller ~]# cat ns-quota-limit.yml
apiVersion: v1
kind: ResourceQuota
metadata:
name: resource-quota
namespace: quota-example
spec:
hard:
limits.cpu: 1
Next apply this YAML to the newly created NS earlier:
[root@controller ~]# kubectl apply -f ns-quota-limit.yml
resourcequota/resource-quota created
Verify the allocated quota:
[root@controller ~]# kubectl describe ns quota-example
Name: quota-example
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: resource-quota
Resource Used Hard
-------- --- ---
limits.cpu 0 1
No LimitRange resource.
Now we will create a simple example pod with nginx image and assign a
CPU resource limit of 500m. So with this we should be allowed to
create upto two Pods only because after that that CPU quota limit
would be reached.
[root@controller ~]# cat pod-nginx-lab-1.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: example
namespace: quota-example
spec:
selector:
matchLabels:
app: example
template:
metadata:
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
cpu: 1100m
We are using Deployment kind so that we can scale the number of Pods
to verify our quota limit. Let’s create this deployment:
[root@controller ~]# kubectl create -f pod-nginx-lab-1.yml
deployment.apps/example created
Once you have submitted the deployment manifest to Kubernetes with
kubectl, check that the pod is running:
[root@controller ~]# kubectl get pods -n quota-example -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-787448d859-5q7dp 0/1 ContainerCreating 0 2s <none> worker-2.example.com <none> <none>
Now, scale up the deployment and observe that additional pods are created:
[root@controller ~]# kubectl -n quota-example scale deployment/example --replicas=5
deployment.apps/example scaled
Because we specified a CPU limit of 500m, there is no problem scaling our deployment to two replicas, which uses the two cores that we specified in our quota.
[root@controller ~]# kubectl get pods -n quota-example -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
example-787448d859-5q7dp 1/1 Running 0 31s 10.44.0.2 worker-2.example.com <none> <none>
example-787448d859-pk8d5 1/1 Running 0 62s 10.36.0.2 worker-1.example.com <none> <none>
But if you now try to scale the deployment so it uses more resources than specified in the quota, you will find that additional pods are not scheduled by Kubernetes:
[root@controller ~]# kubectl -n quota-example get events
....
LAST SEEN TYPE REASON OBJECT MESSAGE
4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-v9qbv" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-wn52k" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
4m57s Warning FailedCreate replicaset/example-1-78b64cfbf5 Error creating: pods "example-1-78b64cfbf5-kp4gn" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=600m, used: limits.cpu=500m, limited: limits.cpu=1
...
To delete the deployment:
[root@controller ~]# kubectl delete deployments -n quota-example example
deployment.apps "example" deleted
We will also delete the namespace:
[root@controller ~]# kubectl delete ns quota-example
namespace "quota-example" deleted
Example: Define a count quota for pods
In this example we will create a namespace with quota limit for the number of pods which can be created within the namespace.
[root@controller ~]# cat pod-quota-limit.yml
apiVersion: v1
kind: ResourceQuota
metadata:
name: pods-quota
namespace: pods-quota-ns
spec:
hard:
pods: 2
Since we are planning to map this quota to a new namespace, we will first need to create a new namespace:
[root@controller ~]# kubectl create ns pods-quota-ns
namespace/pods-quota-ns created
Now we can create a new quota and apply it to this new created namespace:
[root@controller ~]# kubectl apply -f pod-quota-limit.yml
resourcequota/pods-quota created
Verify the namespace is created successfully:
[root@controller ~]# kubectl get ns pods-quota-ns
NAME STATUS AGE
pods-quota-ns Active 2m6s
To check the resource quota limit which is applied to pods-quota-ns
namespace:
[root@controller ~]# kubectl get resourcequota -n pods-quota-ns
NAME AGE REQUEST LIMIT
pods-quota 2m34s pods: 0/2
We can also use kubectl describe to get more details on the quota
limit on respective namespace:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 0 2
No LimitRange resource.
So, we have defined a hard limit of 2 pods in our pods-quota-ns
namespace. Let us create a deployment to make sure, more than 2 pods are
not created in this namespace:
[root@controller ~]# cat nginx-example.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-1
namespace: pods-quota-ns
spec:
selector:
matchLabels:
run: nginx
template:
metadata:
labels:
run: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
Let us create this deployment:
[root@controller ~]# kubectl create -f nginx-example.yml
deployment.apps/nginx-1 created
Our pod has successfully started:
[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME READY STATUS RESTARTS AGE
nginx-1-658f4cf99f-j2qkb 1/1 Running 0 23s
Now we will scale the number of pods in our deployment to 5 and check if those are created:
[root@controller ~]# kubectl -n pods-quota-ns scale deployment/nginx-1 --replicas=5
deployment.apps/nginx-1 scaled
List the available pods in pods-quota-ns namespace:
[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME READY STATUS RESTARTS AGE
nginx-1-658f4cf99f-j2qkb 1/1 Running 0 18m
nginx-1-658f4cf99f-jhfx2 1/1 Running 0 20s
As you see, even though we gave the scale limit as 5, only 2 pods were
created as we have a hard quota limit on number of pods allowed in
pods-quota-ns to be maximum as 2.
You can verify the events in this namespace:
[root@controller ~]# kubectl -n pods-quota-ns get events
...
2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-rksfk" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-567pj" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m59s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-mccth" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
2m58s Warning FailedCreate replicaset/nginx-1-658f4cf99f Error creating: pods "nginx-1-658f4cf99f-z79vh" is forbidden: exceeded quota: pods-quota, requested: pods=1, used: pods=2, limited: pods=2
...
The same can be checked using kubectl describe command where the used
quota is same as hard limit so no more pods will be allowed to be
created in this NS:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 2 2
No LimitRange resource.
Understanding Limit Range
When you are using quotas on a namespace, one requirement is that every container in the namespace must have resource limits and requests defined. Sometimes this requirement can cause complexity and make it more difficult to work quickly with Kubernetes. Kubernetes provides the facility for default requests and limits to be provided at the namespace level. You could use this to provide some sensible defaults to namespaces used by a particular application or team.
We can configure default limits and requests for the containers in a
namespace using the LimitRange object. This object allows us to
provide defaults for the CPU or memory, or both. If a LimitRange
object exists in a namespace, then any container created without the
resource requests or limits configured in LimitRange will inherit these
values from the limit range.
There are two situations where LimitRange will affect the resource
limits or requests when a pod is created:
- Containers that have no resource limits or requests will inherit the
resource limit and requests from the
LimitRangeobject - Containers that have no resource limits but do have requests specified will inherit the resource limit from the LimitRange object
If a container already has limits and requests defined, then
LimitRange will have no effect. Because containers that specify
only limits default the request field to the same value, they will not
inherit the request value from LimitRange.
Example: Create and apply LimitRange to namespace
In this example we will create a LimitRange and assign it to our
existing namespace pods-quota-ns where we had assigned a quota limit.
[root@controller ~]# cat assign-limit-range.yml
apiVersion: v1
kind: LimitRange
metadata:
name: define-limit
namespace: pods-quota-ns
spec:
limits:
- default:
memory: 512Mi
cpu: 1
defaultRequest:
memory: 256Mi
cpu: 500m
type: Container
Let’s create this LimitRange:
[root@controller ~]# kubectl create -f assign-limit-range.yml
limitrange/define-limit created
To get the list of LimitRange in pods-quota-ns namespace:
[root@controller ~]# kubectl get limits -n pods-quota-ns
NAME CREATED AT
define-limit 2020-11-29T07:15:18Z
We can verify the LimitRange using kubectl describe:
[root@controller ~]# kubectl describe ns pods-quota-ns
Name: pods-quota-ns
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: pods-quota
Resource Used Hard
-------- --- ---
pods 2 2
Resource Limits
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu - - 500m 1 -
Container memory - - 256Mi 512Mi -
Instead of using describe with namespace, we can use it with
LimitRange directly:
[root@controller ~]# kubectl describe limits -n pods-quota-ns define-limit
Name: define-limit
Namespace: pods-quota-ns
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu - - 500m 1 -
Container memory - - 256Mi 512Mi -
Now let’s create a new deployment without specifying the resource limits (but before that I will delete the existing deployment which I created earlier):
[root@controller ~]# kubectl delete deployment -n pods-quota-ns nginx-1
deployment.apps "nginx-1" deleted
[root@controller ~]# kubectl get pods -n pods-quota-ns
NAME READY STATUS RESTARTS AGE
nginx-1 1/1 Running 0 25s
Now let’s check the details of this pod to make sure if the default resource limit is applied:
[root@controller ~]# kubectl describe pods -n pods-quota-ns nginx-1
Name: nginx-1
Namespace: pods-quota-ns
Priority: 0
Node: worker-1.example.com/192.168.43.49
Start Time: Sun, 29 Nov 2020 13:03:22 +0530
Labels: run=nginx-1
...
State: Running
Started: Sun, 29 Nov 2020 13:03:34 +0530
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 512Mi
Requests:
cpu: 500m
memory: 256Mi
...
As expected, even when we didn’t provided any Resource Limits to this
Pod, the pod has default LimitRange applied.
To delete the LimitRange we created in pods-quota-ns namespace:
[root@controller ~]# kubectl delete limitranges -n pods-quota-ns define-limit
limitrange "define-limit" deleted
Conclusion
In this Kubernetes tutorial we learned how we can define resource quota to namespace to handle resource scarcity across Cluseter nodes. If your namespace has a resource quota, it is helpful to have a default value in place for CPU limit. Here are two of the restrictions that a resource quota imposes on a namespace:
- Every Container that runs in the namespace must have its own CPU limit.
- The total amount of CPU used by all Containers in the namespace must not exceed a specified limit.
If a Container does not specify its own CPU limit, it is given the default limit, and then it can be allowed to run in a namespace that is restricted by a quota.
Also Read (External):
Configure
Minimum and Maximum CPU Constraints for a Namespace
Configure
Default CPU Requests and Limits for a Namespace
