Releases: kubernetes-sigs/kubespray
Releases · kubernetes-sigs/kubespray
v2.24.1
Changes by Kind
Feature
Bug or Regression
- Add configuration to create cilium CNI plugin file when cilium>=1.14.0 (#10945, @cleman95 )
- Fix logical error when checking for boostrap-os (#10953, @VannTen)
- Make containerd 1.7.13 default
Make runc 1.1.12 default
Patch GHSA-xr7r-f8xq-vfvv (#10877, @VannTen)
Other (Cleanup or Flake)
The release intend to address GHSA-xr7r-f8xq-vfvv
v2.22.2
Changes by Kind
Network
API Change
Feature
- Add hashes for kubernetes version 1.26.6, 1.26.7, 1.26.8 & 1.26.9 (#10444, @bozzo)
- Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
- Make kubernetes 1.26.13 the default version (#10823, @VannTen)
Failing Test
Bug or Regression
- Fix hardcoded pod infra version (#10805, @ErikJiang)
- Make containerd 1.7.13 default
Make runc 1.1.12 default
Patch GHSA-xr7r-f8xq-vfvv (#10878, @VannTen) - [Multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)
The release intend to address GHSA-xr7r-f8xq-vfvv
v2.23.3
Changes by Kind
Feature
Bug or Regression
- Fix hardcoded pod infra version (#10806, @ErikJiang)
- Make containerd 1.7.13 default
Make runc 1.1.12 default
Make kubernetes 1.27.10 default
Patch GHSA-xr7r-f8xq-vfvv (#10876, @VannTen)
Other (Cleanup or Flake)
The release intend to address GHSA-xr7r-f8xq-vfvv
v2.24.0
Deprecation / Removal
- Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (#10464, @unai-ttxu)
- Drop support for Kubernetes 1.25.x (move min version to 1.26.x) (#10420, @yankay)
- Drop installation notes for Debian Jessie (#10642, @jelmer)
Feature / Major Changes
- Make kubernetes v1.28.6 default (#10810, @mzaian)
- Add kubernetes v1.28.0, v1.28.1, v1.28.2, v1.28.3, v1.28.4, v1.28.5 hash (#10435, #10541, #10739, @mzaian ; #10390, @tmurakam ; #10624, @tmurakam)
- Add Retry for Applying PriorityClass (#10469, @hangscer8)
- Add option
crio_criu_support_enabled
to enable container forensic analysis (#10479, @tu1h) - Add option
kubectl_alias
to set bash alias of kubectl (#10552, @tu1h) - Add variable to configure ipvs modules (
kube_proxy_ipvs_modules
) (#10580, @borgiacis) - Check nameserver only when dns is enable (#10561, @yckaolalala)
- Correctly handle remove_default_searchdomains when value is undefined (#10533, @yckaolalala)
- Kube-scheduler: remove/update deprecated component component config v1beta3. (#10484, @mzaian)
- Terraform-aws: variable driven ami selection (
ami_name_pattern
/ami_virtualization_type
/ami_owners
) (#10520, @mertcancam) - Terraform-openstack: Added possibility to enable dhcp flag critical on one interface (#10446, @Xartos)
- This will introduce a new variable
kube_apiserver_admission_plugins_podnodeselector_default_node_selector
that can be used withkube_apiserver_admission_plugins_needs_configuration: [PodNodeSelector]
defined. So allows the users to configure PodNodeSelector plugin. (#10607, @titansmc) - UpCloud: Terraform provider updated to v2.12.0. Server groups with strict anti-affinity (move var from
anti_affinity_policy
toanti_affinity
) (#10474, @robinAwallace) - Update dockerfile to follow best practices (#10708, @maxime1907)
- Update to ansible 2.15 and set minimum version to 2.15.5 (#10481, @MrFreezeex)
- [etcd] Update Default etcd version to 3.5.10 for kubernetes 1.28, 1.27 and 1.26 (#10798, @VannTen)
- [etcd] update version to 3.5.9 for k8s 1.28 , 1.27 , 1.26 (#10482, @mzaian)
- [etcd] add 3.5.10 hashes (#10566, @mzaian)
- [vsphere_csi] Update to 3.1.0 supports Kubernetes Version 1.28 (#10451, @mzaian)
- [cinder_csi] Cinder-CSI now use
cluster_name
variable instead of the default hardcoded "kubernetes" value (#10422, @floryut)
Applications
- [argocd] update argocd to v2.8.4 (#10568, @mzaian)
- [helm] upgrade to 3.13.1 (#10567, @mzaian)
- [coredns] Added option coredns_additional_error_config to allow for configuration of the coredns error plugin. (#10501, @Elias-elastisys)
- [coredns] Support CoreDNS use host network & config CoreDNS port (#10617, @liuxu623)
- [coredns] Support disable dns autoscaler when use CoreDNS (#10608, @liuxu623)
- [coredns] Add pdb to coredns (#10557, @lobiyedKarim1)
- [cert-manager] upgrade to v1.13.2 (#10616, @liuxu623)
- [cert-manager] Upgrade to v1.12.6 (#10582, @chansuke)
- [cert-manager] Upgrade to v1.12.5 (#10500, @chansuke)
Network
- [cilium] Fix invalid hubble yaml if
cilium_hubble_tls_generate
is enabled (#10430, @toonalbers) - [cilium] Use correct ports in cilium metrics services if metrics are enabled. (#10519, @bakito)
- [cilium] Adds support for deploying clusters with cilium 1.14+ (#10684, @rl0nergan)
- [calico] Separate calico-node and calico-cni-plugin service accounts and update default calico to v3.26.1 (#10416, @mzaian)
- [calico] Use calico_pool_blocksize from cluster when existing (#10516, @VannTen)
- [calico] Update default calico to v3.26.3 (#10526, @mzaian)
- [calico] Update default calico to v3.26.4 (#10669, @mzaian)
- [kube-router] Default kube-router version updated to v2.0.0 (#10503, @bozzo)
- [kube-router] Default kube-router version updated to v1.6.0 (#10478, @bozzo)
- [kube-router] Add
kube_router_bgp_graceful_restart
optional setting for disabling graceful BGP restarts (default to true) (#10489, @rosskusler) - [metallb] Add option to set avoidBuggyIPs in IPAddressPools and change the default back to false (#10458, @zeeZ)
- [metallb] Metallb --lb-class cmd arg to support multiple LoadBalancer implementations (#10550, @Seal1998)
- [custom_cni] Add helm support for custom_cni deployment (#10529, @kukacz)
- [kube_vip] Add
kube_vip_lb_fwdmethod
option for kube-vip (#10762, @tu1h)
Container-Managers
- [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)
- [containerd] Make containerd 1.7.11 default (#10671, @mzaian)
- [containerd] Add hashes for containerd versions 1.7.6 ~ 1.7.8 default (#10439, #10525, #10589, @mzaian)
- [containerd] Specify the runc path when we use the containerd container engine and change the bin_dir path. (#10154, @qlijin)
- [containerd] Refactor NRI activation for containerd and CRI-O (remove
crio_enable_nri
andcontainerd_nri_disable
) now only one varnri_enabled
default to false (#10470, @fmuyassarov) - [containerd] Add Boolean option
enable_cdi
to enable cdi (false by default) (#10603, @krembu) - [containerd] Add configuration option for NRI (disable by default) in crio & containerd (using new
containerd_nri_disable
andcrio_enable_nri
) (#10454, @fmuyassarov) - [containerd] add config support
override_path
(#10776, @yankay) - [runc] Upgrade to v1.1.10 (#10671, @mzaian)
- [crio] Update to v1.28.1 (#10480, @qlijin)
- [crio] Remove crio package configuration during cleanup (#10584, @yckaolalala)
- [crio] Update docs for crio_registry_auth (#10785, @qlijin)
- [docker] Ability to define GPG key path for Docker APT (using new variable
docker_repo_key_keyring
) (#10513, @emiran-orange) - [kata-containers] Freshens configuration-qemu to latest template compatible with kata-containers 3.1.3. (#10466, @Alphadelta14)
- [nerdctl] Bump nerdctl version 1.7.1 (#10685, @yankay)
- [nerdctl] Change nerdctl version from 1.5.0 to 1.6.0 (#10475, @MaGaroo)
Documentation
- Add link to Cilium CNI documentation (#10431, @toonalbers)
- Update docs for calico_iptables_backend in Redhat/Centos.md (#10417, @yankay)
- Update metallb example configs (#10485, @caruccio)
- Updated AWS ALB ingress controller version (#10680, @kundan2707)
Bug or Regression
- Add a variable reset_restart_network_service_name in the reset role to be able to configure the name of the service which is restarted. (#10428, @RomainMou)
- Add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true (#10618, @Payback159)
- Check for correct conntrack module presence, regardless of kernel versions (#10662, @VannTen)
- Fallback_ips: ignore unreachable hosts (#10601, @poblahblahblah)
- Fix 'kube-apiserver' tag inappropriately overwriting secrets at rest encryption token (#10460, @jwitko)
- Fix assertion for task item verify-settings (#10699, @piwinkler)
- Fix external-lb in kubelet.conf server address and kube-proxy api-server address (#10490, @ugur99)
- Fix forgotten update of etcd-servers list in apiserver manifest when scaling (#8253, @liupeng0518)
- Fix metallb example yaml (#10545, @caruccio)
- Fix reset job for cri-o container engine (#10197, @turbosnail)
- Fix restart network task cannot be skipped (ansible boolean conversion needed) (#10512, @ErikJiang)
- Fix: add kubelet tag in task of Fetch facts to avoid kubelet config inconsistencies (#10423, @NierYYDS)
- Fixes the path of the certificates use in the etcdctl.sh wrapper when the deployment type is not kubeadm. (#10467, @RomainMou)
- Hubble relay will work when cilium_cluster_name is customised. (#10614, @eugene-eeo)
- Disable podCIDR allocation from control-plane when using calico (#10639, @VannTen)
- Kubespray-defaults: Check for boostrap-os FQDN (#10590, @VannTen)
- Patch for modprobe_nf_conntrack for new Linux Kernel, when using ipvs (#10625, @abhishekkr)
- Remove always tag applied on bootstrap (#10556, @yckaolalala)
- Set remove_default_searchdomains to false by default (#10554, @hedayat)
- Swap is now disabled using systemd (mask of swap.target) (#10587, @VannTen)
- Fix undefined retries variable when copying etcdctl (#10634, @ErikJiang)
- Move control plane certs renewal "spread out" into the systemd timer (#10596, @VannTen)
- The dhcp configuration for dns nameservers are now the same than during installation (#10548, @smutel)
- Use correct env var name for kube-vip per service leader election (#10433, @ThisIsQasim)
- Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
- Fix download retry when get_url has no status_code (#10613, @RomainMou)
- Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
- Set the
maxUnavailable
of the coredns rolling update strategy to 1 (#10748, @tu1h) - Fix crio_version version comparison (#10780, @ledroide)
- Fix disable swap failed in Centos/RHEL 7 (#10751, @yankay)
- Fix image pull fail with insecure-registry (#10775, @yankay)
- Refactor check_galaxy + fix version (#10729, @VannTen)
- Fix Helm installation on SLES and openSUSE (#10794, @goldyfruit)
- Fix incorrect ciliumcli binary (#10575, @tu1h)
- Fix ntp installation on SLES and openSUSE (#10786, @goldyfruit)
- Fix the cluster installation on cluster using etcd clients nodes (cilium / calico / ...) (#10769, @VannTen)
Other (Cleanup or Flake)
- Cleanup a deprecation warning (ipaddr filter) (#10518, @VannTen)
- Decouple kubespray-defaults from download (#10626, @VannTen)
- Etcd/backup: use native ansible modules instead of shell (#10540, @VannTen)
- Etcd: use dynamic group for certs generation check (#10610, @VannTen)
- Factorize some identical playbooks steps into their own sub-playbooks (#10633, @VannTen)
- Pre-upgrade tasks cleanup (#10656, @VannTen)
- Refactor "multi" handlers to use listen (#10542, @VannTen)
- Remove unneeded workaround for removing kubeadm DNS (#10695, @VannTen)
- Removed DEPRECATED
--logtostderr
from metrics-server (#10709, @micha...
v2.23.2
Container-Managers
- [containerd] Fix invalid version check in containerd jinja-template config (#10620, @khanhngobackend)
API Change
Feature
- Don't fail on 304 Not Modified for an already downloaded file (#10452, @sathieu)
- Update kubernetes default version to 1.27.9
- Update etcd version for 1.27 and 1.26 to 3.5.10 (#10797, @VannTen)
Failing Test
Bug or Regression
- Fix calico-node in etcd mode. (#10768, @VannTen)
- Fix download retry when get_url has no status_code (#10613, @RomainMou) (#10791, @VannTen)
- Kube-controller-manager will no longer assign pod CIDRs to cluster nodes when using calico (with its default IPAM, calico_ipam_host_local now has a default value of
false
) [⚠️ NOTE users using a non-true value for calico_ipam_host_local will need to change it totrue
] (#10639, @VannTen)
Other (Cleanup or Flake)
v2.23.1
Network
- [Cilium] Fix invalid hubble yaml if
cilium_hubble_tls_generate
is enabled (#10476, @toonalbers)
Feature
- Add hashes for kubernetes 1.27.6 & 1.26.9 (#10443, @bozzo)
- Make kubernetes v1.27.7 default (#10543, @mzaian)
- [etcd] Default version to 3.5.9 for k8s 1.25 , 1.26 , 1.27 (#10483, @mzaian)
- Add crictl 1.26.1 for Kubernetes v1.26 (#10562, @mzaian)
- Change default cri-o versions for Kubernetes 1.25, 1.26 (#10563, @mzaian)
- [ingress-nginx] Fix nginx controller leader election RBAC permissions (#10569, @mzaian)
- Refactor NRI activation for containerd and CRI-O (remove
crio_enable_nri
andcontainerd_nri_disable
) now only one varnri_enabled
default to false (#10496, @fmuyassarov)
Bug or Regression
- Fix get currently configured nameservers error where there are inline comments in /etc/resolv.conf (#10415, @yankay)
- Migrate node-role.kubernetes.io/master to node-role.kubernetes.io/control-plane (#10532, @unai-ttxu)
- [download] Don't fail on 304 Not Modified (#10559, @RomainMou)
v2.23.0
Deprecation / Removal
- Ubuntu 16 and 18 are no longer tested (#10107, @MrFreezeex)
- Drop support for ansible-core 2.11 and update tests dependencies (#10034, @MrFreezeex)
- Drop Kubernetes 1.24 support (#10234, @MrFreezeex)
Feature / Major Changes
- Make kubernetes v1.27.5 default (#10392, @mzaian)
- Add kubernetes v1.27.4 (#10359, @mzaian)
- Add Kubernetes 1.27.2 (#9976, @mzaian)
- Add hashes for 1.27.3 1.26.6, 1.25.11 (#10220, @mzaian)
- Add hashes for 1.27.4 1.26.7, 1.25.12 (#10300, @mzaian)
- Add CPU Management Policies on the Node (#10309, @yankay)
- Add Debian 12(bookworm) support (#10221, @tu1h)
- Add
download.timeout
to update download timeout value (#10149, @yjqg6666) - Add corresponding coredns versions to all the supported kubernetes releases. (#10233, @mzaian)
- Add growpart azure enabled (#10241, @pedro-peter)
- Add ingressClass resource for ingress_nginx by default (#10091, @peschmae)
- Add kubelet topology manager policy on the node (
kubelet_topology_manager_scope
andkubelet_topoloy_manager_policy
) (#10370, @tu1h) - Add labels to kube-vip static pods (#10139, @liupeng0518)
- Add node_taints to aws_inventory script (#10170, @mstoetzer)
- Add option to set
SSL_CERT_FILE
for offline installation using custom CA for https proxy (#10215, @HappyFX) - Add terraform support for NIFCLOUD (#10227, @ystkfujii)
- Add the huawei cloud controller as external cloud controller (#10198, @dabeck)
- Show detected ansible version when it isn't compatible with kubespray (#10109, @jcpunk)
- Allow to override etcd listen-metrics-urls configuration (using
etcd_listen_metrics_urls
variable) (#10332, @forselli-stratio) - Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
- Permit custom names for API server lb/proxy containers (#10166, @jcpunk)
- Permit skipping helm update (#10169, @jcpunk)
- Split defaults main file into 2 files (checksums and version) (#10121, @electrocucaracha)
- System upgrade for Debian-family nodes is available with system_upgrade=true (#10184, @sathieu)
- Update download_hash.sh script (#10120, @electrocucaracha)
- Use a uniform way to get the local path of the binaries (#10211, @ErikJiang)
- Disable fapolicyd service (#10081, @epif4nio)
- Upgrade the load balancer ( nginx and haproxy ) image version to Nginx 1.25, Haproxy 2.8. (#10409, @yankay)
- [etcd] Default version to 3.5.7 for kubernetes 1.27 (#10410, @mzaian)
Applications
- [argocd] update argocd to v2.7.4 (#10226, @mzaian)
- [argocd] update argocd to v2.8.0 (#10364, @mzaian)
- [argocd] Add argocd_install_url option to allow changing argocd url (#10176, @liupeng0518)
- [helm] upgrade to 3.12.1 (#10225, @mzaian)
- [helm] upgrade to 3.12.3 (#10365, @mzaian)
- [helm] add python dependency check for helm-apps (#10192, @palmeXx)
- [krew] add krew_no_upgrade_check (#10175, @liupeng0518)
- [coredns] Bump coredns version to 1.10.1 (#10199, @eminaktas)
- [coredns] Bump nodelocaldns version to 1.22.20 (#10200, @eminaktas)
- [cert-manager] This introduces a new variable for the cert-manager implementation that will allow one to pass in extra arguments to the cert-manager controller.(#10049, @phunyguy)
- Update Helm (v3.12.2) / Skopeo (v1.13.0) and yq (v4.34.2) (#10295, @tu1h)
- Upgrade many tool versions (Helm, crun, kata, youki, gvisor, skopeo, Calico, Cilium etc...) (#9798, @electrocucaracha)
- [local_path_provisioner] Fix invalid podhelper yaml (#10237, @MrFreezeex)
- Update metrics server to v0.6.4 (#10400, @mzaian)
Container-Managers
- [containerd] Make containerd 1.7.5 default (#10397, @mzaian)
- [containerd] Support containerd v1.7.2 (#10219, @Dentrax)
- [containerd] Support containerd 1.7.3 (#10368, @mzaian)
- [containerd] containerd config_path enable mirrors config using new variable
containerd_registries_mirrors
(deprecate and removecontainerd_insecure_registries
for containrd andnerdctl_extra_flags
andinsecure_registry
setting for nerdctl (#10196, @yckaolalala) - [crio] Add crio_insecure_registries option for specifying insecure_registries of crio (#10142, @qlijin)
- [crio]
runroot
now needs to be setup in storage.conf instead of crio.conf (#10372, @floryut) - [crio] Fix etcdctl copy operation (#10242, @ErikJiang)
- [Kata] Set/keep owner/group root/root when unarchiving kata-containers (#10338, @rybnico)
- [youki] Fix youki binary download url (not requiring 'v' in version) (#10337, @ErikJiang)
Network
- [calico] Use configmap to configure calico cni config (#10177, @cyclinder)
- [calico] Update calico v3.25.2 (#10414, @mzaian)
- [calico] Add calico version to v3.26.0 (#10224, @mzaian)
- [calico] Add calico version to v3.26.1 (#10235, @mzaian)
- [calico] Clean up calicoctl_alternate_download_url and calicoctl.mirrors (#10271, @yckaolalala)
- [cilium] Add custom rules to clusterrole for cilium operator (#10267, @jeremythuon)
- [cilium] Upgrade to version 1.13.4 (#10269, @yulng)
- [Cilium] Do not mount tls when 'cilium_hubble_tls_generate' is false (#10357, @charlychiu)
- [Cilium] Update cilium to 1.13.3 (#10158, @jcpunk)
- [flannel] Only create /var/lib/calico when needed (#10156, @jcpunk)
- [flannel] Bump flannel version to v0.22.0 and flannel-cni-plugin version to v1.1.2. Also, changes flannel repository from flannelcni to flannel (#10205, @eminaktas)
- [flannel] Remove unused flannel_cni_download_url (#10188, @oomichi)
- [kube-ovn]: update version v1.11.5 (#10125, @yankay)
- [multus] Fix loop_control template error when item is None (#10347, @nicolas-goudry)
API Change
- Unless the pod security standard versions are changed on intentionally, as default it will be the same major version with Kubernetes version. (#10210, @ugur99)
- Upgrade ansible to 7.0 and ansible-core to 2.14.x (#10190, @MrFreezeex)
⚠️ (See Notes 2)
Documentation
- Add github container registry (
github_image_repo
) to docs/offline-environment.md (#10265, @blackliner) - Update doc for ansible-core 2.14 support and clarify issues running older python versions (#10261, @MrFreezeex)
- Update links for aws_alb_ingress_controller (#10264, @kundan2707)
- Update links in ingress-controller and kuberentes-apps (#10239, @vaibhav2107)
- Update Calico to lowercase and fix broken calico link in README (#10232, @Xieql)
- Document containerd command to restart nginx-proxy container when adding control plane node (#10406, @nicolas-goudry)
Failing Test
- Increase metallb wait timeout from 30sec to 2min (#10260, @MrFreezeex)
- Update CentOS 7 image and test fedora 37 and 38 instead of fedora 35 and 36 (#10108, @MrFreezeex)
Bug or Regression
- Fix Dockerfile for newest directory layout (#10128, @dabeck)
- Fix Flatcar bootstrap issues (yaml module missing and ntp issue) (#10363, @tenni-paws)
- Fix argocd install not working using the kubespray docker image (#10371, @cortex3)
- Fix correctly mount ssl ca directories (#9794, @maxime1907)
- Fix etcdctl copy operation (#10230, @ErikJiang)
- Fix gce-pd-csi driver (#10208, @ashishsinghdev)
- Fix grep command without -w option causing prefix matched while adding one etcd member (#10291, @yangsenzk)
- Fix hcloud-cloud-controller-manager not working in certain setups (#10297, @cortex3)
- Fix helm (kubelet-csr-approver) installation on redhat distro (#10204, @MrFreezeex)
- Fix kubelet-csr-approver usage with upgrade-cluster.yml and missing package with helm role (#10165, @j4m3s-s)
- Fix nginxingress-class template (missing newline) (#10174, @richard-fairthorne)
- Fix problem migration problem with k8s 1.27 (#10136, @batazor)
- Fix reset_confirmation not working when inputing correct value (#10288, @somewho)
- Fix wrong path in manage-offline-files script (#9886, @Medosopher)
- Fix an issue where using Rocky Linux 8 as OS for Vagrant for testing purposes causing etcd to fail on start. (#10252, @nltimv)
- Fix ansible-lint galaxy rule (#10277, @MrFreezeex)
- Fix ansible-lint key-order error (#10314, @MrFreezeex)
- Fix outdated tag and experimental ansible-lint rules (#10254, @MrFreezeex)
- Fix dockerfile build error (#10127, @yankay)
- Fix metrics-server deployment to run with kubernetes 1.26+ (#10183, @mzaian)
- Fix undefined
reset_confirmation_prompt
variable in reset play (#10303, @Mishavint) - Fix CIS Kubernetes V1.23 Benchmark item number 4.1.9 to enhance security (Change kubelet-config.yaml and kubelet.env file permissions from 640 to 600) (#10304, @satandyh)
- Fix parsing of RHSM proxy configuration (#10228, @tmurakam)
- Fix var-spacing ansible rule (#10266, @MrFreezeex)
- Fix specify owner to kube_owner in task of copy cni plugins (#10407, @NierYYDS)
- Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8)
- Fix recover_control_plane playbook (also add debian 12 with cilium as a new nightly test) (#10411, @floryut)
- Fix nameserver inline comments in /etc/resolv.conf (#10415, @yankay)
- Added
systemd_resolved_disable_stub_listener
variable to disable systemd-resolved's stub listener, defaults totrue
on Flatcar. (#9875, @cosandr) - Remove
auto_attach
andsyspurpose
in RHEL subscription Organization ID/Activation Key registration. (#10258, @yckaolalala) - Replace "crio_packages" with "crio_bin_files" (#10182, @yckaolalala)
- Update MetalLB deployment, wait for resource. (#9995, @Jeroen0494)
- Upgrade ansible to 7.0 and ansible-core to 2.14.x in Dockerfile (#10259, @yckaolalala)
- Fix typo kubelet_topoloy_manager_policy => kubelet_topology_manager_policy (#10384, @hangscer8)
⚠️ (See Notes 1) - Change maximal_ansible_version to 2.15(exclusive) (#10395, @yankay)
- Install etcdutl file by default (#10385, @liupeng0518)
Other (Cleanup or Flake)
v2.22.1
Bug or Regression
- Don't let find search filesystem mounts in docker build run step (#10131, @tomodachi)
- Fix Dockerfile for newest directory layout (#10128, @dabeck)
- Fix dockerfile build error (#10181, @yankay)
- Fix metrics-server deployment to run with kubernetes 1.26+ (#10183, @mzaian)
- update README for v2.22.0 (#10180, @Payback159)
- Fix Update MetalLB deployment, wait for resource. (#9995, @Jeroen0494)
v2.22.0
Deprecation / Removal
- [Cilium] Delete the probe option of cilium_kube_proxy_replacement (#9929, @XiuguangHuang)
- [Cilium] Remove use_localhost_as_kubeapi_loadbalancer and detect wether we can use localhost apiserver loadbalancer if cilium/calico replace kube-proxy (#9718, @MrFreezeex)
- Drop
crun_bin_dir
unused variable, now using onlybin_dir
var (#9845, @electrocucaracha) - Drop the canal network_plugin support because the network_plugin is unmaintained. (#10100, @oomichi)
- Remove the support of Debian 9 (#10097, @yankay)
- Replaces storage.googleapis.com/kubernetes-release with dl.k8s.io (#10066, @KlwntSingh)
Feature / Major Changes
- Add Kubernetes 1.26.x (#9570, @mzaian ; #9732, @yankay; #9829, @mzaian; #9900, @mzaian)
- Make kubernetes v1.26.5 default (#9983, @mzaian)
- "native" snapshotter of nerdctl config is replaced by new var
nerdctl_snapshotter
with default "overlayfs" value (#9979, @dmitrytretyakov) - Support multi-arch using the same image name (#9978, @ErikJiang)
- Add DNS configuration for cert-manager (using new variables
cert_manager_dns_policy|config
) (#9673, @ErikJiang) - Add Retry for restart kube-controller-manager (#10013, @hangscer8)
- Add
coredns_additional_configuration
variable to define extra Coredns configurations (#10025, @navidnabavi) - Add
coredns_rewrite_block
to perform internal message rewriting (#10045, @maxime1907) - Add a new simple network_plugins custom_cni to install user provided manifests (#9819, @MrFreezeex)
- Add back openssh-client to docker image (#9835, @maxime1907)
- Add download retries option
download_retries
(#9911, @tu1h) - Add support to install ContainerD on any Linux Distributions using new var
allow_unsupported_distribution_setup
(#9827, @XDRAGON2002) - Add the
kube-profile
config to the kubeadm'skube-scheduler
config. (#9993, @yankay) - Add vim to kubespray docker image (#9805, @XDRAGON2002)
- Adds support for Kubelet-CSR-approver to auto-approve kubelet CSR when
kubelet_rotate_server_certificates
. (#9877, @j4m3s-s) - Add
dns_cpu_limit
value to support large scaled coredns deployments (#10103, @mzaian) - Add provider meta module_name in Equinix Metal TF configs (#10044, @Vasubabu)
- Allow to configure image garbage collection (using
kubelet_image_gc_high_threshold
andkubelet_image_gc_low_threshold
) (#9832, @zhan9san) - Apply kubeadm patches during upgrade as recommended by k8s (#9781, @mvandergiesen)
- Cinder-csi: Allow VolumeSnapshotClass' deletionPolicy to be configurable (#9736, @huangkevin404)
- Containerd add
containerd_use_config_path
config field. (#9770, @lengrongfu) - Enable control plane load balancing for kube-vip (#9785, @ErikJiang)
- Feat(contrib/terraform): support custom ssh port (#9836, @maxime1907)
- Fix kube-bench 1.2.20 to enhance security (Ensure that the --audit-log-maxbackup argument is set to 10) (#9939, @yankay)
- Fix kube-bench 1.1.19 to enhance security (Change Kubernetes Cert directory and file ownership is set to root:root) (#9937, @yankay)
- Fix kube-bench 4.1.1 to enhance security (Change kubelet systemd init file from 644 to 600) (#9934, @yankay)
- Fix kubernetes-app/argocd: download related things with the download role (#9786, @pli01)
- Kube.py now supports kubeconfig (#9982, @liupeng0518)
- MetricsServer: Add extras nodeselector, affinity, tolerations (using
metrics_server_nodeselector
,metrics_server_extra_affinity
,metrics_server_extra_tolerations
) (#9972, @pli01) - Refactor Hetzner terraform (fixing flatcar configs and remove deprecated provider) (#10002, @ThisIsQasim)
- Support for MetalLB v0.13.9 with CRD (#9120, @Jeroen0494)
- Throw an error when specifying unsupported os in Vagrant (#9965, @THUzxj)
- Update CoreDNS manifests (remove deprecated annotations) (#9977, @mzaian)
- Update dns-autoscaler configuration and remove deprecated annotations (#9996, @mzaian)
- Update metrics server to v0.6.3 (#10026, @mzaian)
- Upgrade argocd to v2.6.3 (#9848, @panguicai008)
- Upgrades the following Python libraries to their latest available releases (cryptography / jinja2 / jmespath / MarkupSafe/ netaddr / pbr / ruamel.yaml / ruamel.yaml.clib) (#9938, @luksi1)
- Add IPv6 listen directive to haproxy if enable_dual_stack_networks (#9674, @yankay)
- Add support for Ansible collections in Kubespray (
⚠️ See notes !) (#9582, @luksi1) - Support mTLS for Hubble and upgrade backend to v0.11.0 (#9959, @jeremythuon)
- Update nodelocaldns to 1.22.18 (#9800, @sathieu)
- Replace
disable_swap
variable withkubelet_fail_swap_on
(#10036, @Manuelraa) - Replace nodelocaldns label to
k8s-app: node-local-dns
(#9745, @stelucz) - Upgrade rancher local-path-provisioner to v0.0.23 (#9855, @panguicai008)
- Use
kube_apiserver_address
variable for advertiseAddress (#9967, @liupeng0518) - Use string for ipv6 forward conf value (#9992, @liupeng0518)
- Update pause image version to v3.9 (#10112, @mzaian)
- Upgrade cni version to v1.3.0 (#10058, @cyclinder)
- [argocd] update argocd to v2.6.7 (#9953, @mzaian)
- [helm] support to 3.11.1 (#9849, @mzaian)
- [helm] support to 3.11.3 (#10022, @mzaian)
- [helm] support to 3.11.2 (#9951, @mzaian)
- [helm] upgrade to 3.12.0 (#10085, @mzaian)
- [UpCloud] Add server group support for vms and target port for loadbalancers (#9831, @robinAwallace)
- [argocd] update argocd to v2.5.10 (#9753, @yanggangtony)
- [cert-manager] Upgrade to v1.11.1 (#9964, @rtsp)
- [flannel] update to v0.21.4 (#10027, @mzaian)
- [nerdctl] support version 1.3.1 (#10024, @mzaian)
- [nerdctl] update to version 1.4.0 (#10119, @mzaian)
Applications
- [kube-vip] Support to v0.5.8 (#9734, @hangscer8)
- [kube-vip] Support kube-vip to v0.5.11 (#9852, @panguicai008)
- [kube-vip] Update default kube-vip to v0.5.12 (#10005, @hangscer8)
- [vSphere-csi] Add resources section to all containers releated to Vsphere CSI driver (#9687, @JRaver)
- [argocd] update argocd to v2.7.2 (#10086, @mzaian)
Container-Managers
- [containerd] Add hashes for containerd version 1.6.19 (#9838, @mzaian)
- [containerd] Add hashes for containerd version 1.6.20 (#9954, @mzaian)
- [containerd] Add hashes for containerd version 1.7.0 (#9892, @mzaian)
- [containerd] Add hashes for containerd versions 1.7.1, 1.6.21 (#10061, @mzaian)
- [containerd] Support version 1.6.16 (#9727, @yanggangtony)
- [cri-o] Bump versions to 1.26.3, 1.25.3, 1.24.5 (#9999, @dkasanic)
- [cri-o] Fix install order -> first runc then crictl (#9780, @mvandergiesen)
- [cri-o] Fix missed double quotes in cri-o config (#10040, @turbosnail)
- [cri-o] Fix CRI-O amd64 v1.26.0 wrong archive checksum (#9872, @panguicai008)
- [cri-o] cri-o restart if config change (#10057, @MrFreezeex)
- [cri-o] Remove deprecated crio_pids_limit (default is now unlimited) (#10056, @j4m3s-s)
- [cri-o] Fix cri-o restart if config change (#10057, @MrFreezeex)
- [runc] Upgrade to v1.1.7 (#10039, @pomland-94)
Network
- [Calico] Add Retry and Ignore Error for Checking calico ready (#9883, @hangscer8)
- [Calico] Add option
calico_kubeconfig_wait_timeout
(#9994, @tu1h) - [Calico] Improve version check command (#9861, @zhan9san)
- [Calico] Optimize the detection of calico existence (#9873, @hangscer8)
- [Calico] Support calico version v3.25.0 (#9860, @cyclinder)
- [Calico] upgrade default calico version to v3.25.1 (#9950, @mzaian)
- [Calico] Add missing ipamconfigs resource in RBAC (#9755, @chaunceyjiang)
- [Calico] Fix installation while applying CRD (#10068, @hangscer8)
- [Calico] Add calico version to v3.24.6 (#10113, @mzaian)
- [Cilium] Add and support v1.13.0 (#9879, @utam0k)
- [Cilium] Fix Hubble relay configuration (#9876, @prashantchitta)
- [Cilium] Fix the configuration of TLS for hubble (#9880, @utam0k)
- [Cilium] Remove duplicates in the configuration of tls for hubble (#9932, @CaMoPeZzz)
- [Cilium] Support version above 1.13.x (#9914, @wbh1)
- [Cilium] Updates hubble certgen arguments (wrong since v0.1.7) (#9856, @XDRAGON2002)
- [Cilium] IPAM uses "Cluster Scope" mode by default. Also add the parameters required for this mode (#9443, @dcwbq)
- [flannel] Update image repo from flannelcni to flannel (#10041, @ErikJiang)
- [multus] fix multus include error (#10105, @darkobas2)
API Change
- Openstack cloud controller manager bind address is now configurable using
external_openstack_cloud_controller_bind_address
(#9958, @dominykasn)
Documentation
- Add a mention for custom_cni in CNI list (#9878, @j4m3s-s)
- ArgoCD no longer uses the pod name as initial password (#9930, @peschmae)
- Drop remaining part for supporting ansible 2.9 and 2.10 (#9842, @oomichi)
- Fix sidebar documentation (#9988, @lijin-union)
- Fixup link in docs/calico.md (#9940, @kundan2707)
- Remove stale contents for cni documention (#9778, @tu1h)
- Reword confusing etcd download url comment when
etcd_deployment=host
(#9686, @tjanson) - Suggest to run reset.yml playbook for first-time users (#9865, @kerryeon)
- Update docker tag to v2.21.0 in README.md (#9802, @Payback159)
- Update link for baremetel consideration (#9944, @kundan2707)
- Add port requirements documentation (#9969, @yankay)
Failing Test
- Update Terraform to 1.3.7 and Vagrant to 2.3.4 (#9699, @floryut)
- [CI] Migrate CI_BUILD_ID to CI_JOB_ID and CI_BUILD_REF to CI_COMMIT_SHA following gitlab upgrade (#10063, @floryut)
Bug or Regression
- Add PSS labels to metallb namespace (#9713, @manzsolutions-lpr)
- Add jmespath back to Dockerfile image (#9697, @floryut)
- Add missing krew_download_url to offline.yml (#9788, @jianse)
- Add proxy_env variable to apt_key cleanup task (#9766, @SamuelBECK1)
- Add rsync in Dockerfile (#9839, @zhan9san)
- Add ruamel.yaml back to Dockerfile image (#9707, @floryut)
- Cleanup MetalLB install following update (#10004, @eugene-marchanka)
- Copy contrib/ to Dockerfile (#9774, @oomichi)
- Downgrade the version of CoreDNS to 1.8.6 for co...
v2.21.0
Deprecation / Removal
Feature / Major Changes
- Add Check resolv.conf is empty to avoid CoreDNS crash (#9502, @yankay)
- Add XDG related Helm paths to be removed from reset tasks (#9561, @emiran-orange)
- Add a parameter (
disable_host_nameservers
) to disable host nameservers (#9357, @eminaktas) - Add an option (
populate_loadbalancer_apiserver_to_hosts_file
) to skip adding load balancer name in the hosts file (#9331, @JRaver) - Add custom options to coredns kubernets plugin (
coredns_kubernetes_extra_opts
) (#9608, @mvandergiesen) - Add docker support for openEuler linux (#9498, @ErikJiang)
- Add support for the OpenEuler Linux (#9494, @ErikJiang)
- Add terraform script for Flatcar Linux on Hetzner (#9618, @florianow)
- Add the ability to define options for DNS upstream servers (using new variable
dns_upstream_forward_extra_opts
) (#9311, @emiran-orange) - Add var (
ingress_nginx_probe_initial_delay_seconds
) for control initialDelaySeconds in ingress-nginx probes (#9405, @zvlb) - Add variable condition snapshot in vSphere CSI (
vsphere_csi_block_volume_snapshot
) (#9429, @yanggangtony) - Add variable in metrics_server deployment (
metrics_server_replicas
) to enable HA mode (#9539, @ugur99) - Change dns upstream condition for nodelocaldns when using
host_resolvconf
(#9378, @unai-ttxu) - Download coredns image to all hosts in k8s_cluster (#9316, @joes)
- Enable check mode in DNS Cleanup tasks (#9472, @emiran-orange)
- Etcd image has the same tag accross multiple archs (#9516, @hangscer8)
- Fix a pre-upgrade node drain rescue task failure when
kube_override_hostname
is set (#9556, @chadswen) - Fix default value for kubelet_secure_addresses (#9355, @willtrnr)
- Provides <kubeadm_init_timeout> to change the timeout of first control-plane initialization (#9617, @tu1h)
- Remove PodSecurityPolicies in MetalLB for kubernetes 1.25 (#9442, @yanggangtony)
- Support Python 3.11 -
ruamel.yaml.clib
need to be updated to 0.2.7 (#9426, @olivierlemasle) - Support customize the additional sysctl variables using
additional_sysctl
(#9351, @yankay) - Support patches field in kubeadm v1beta3 in both InitConfiguration and JoinConfiguration (using new variable
kubeadm_patches
) (#9326, @titaneric) - Switch helm install (from synchronize to copy) to support password authentication (#9343, @ghostloda)
- Update api version for pdb and batch (deprecated in 1.25) (#9369, @yankay)
- Update dashboard image repo to remove arch flag (#9530, @tu1h)
- Update etcd log-level parameter name (new name:
ETCD_LOG_LEVEL
) (#9540, @ErikJiang) - Update local-volume-provisioner to 2.5.0 + add documentation (#9463, @olivierlemasle)
- Update the number of nofile limits in containerd to 65535 (#9507, @ErikJiang)
- Upgrade metrics server to v0.6.2 (#9554, @mzaian)
- Upgrade the load balancer ( nginx and haproxy ) image version. (#9506, @yankay)
- Use kube_apiserver_port variable instead of hard-coding 6443 (#9620, @huangkevin404)
- [etcd] Default version to 3.5.5 for k8s 1.25.x (#9419, @mzaian)
- Update CoreDNS version to v1.9.3 (#9503, @yankay)
- Add the possibility to specify extra domains for the coredns kubernets plugin (using
coredns_kubernetes_extra_domains
) (#9635, @mvandergiesen) - Streamline ansible_default_ipv4 gathering loop (#9281, @rptaylor)
- Update kubernetes dashboard to 2.7.0 (k8s 1.25 support) (#9425, @mzaian)
- Skip retry operation with containerd when etcd installed on host VM (#9560, @JRaver)
- Update pause image version to v3.8 (#9668, @mzaian)
- Enable kubelet_authorization_mode_webhook back by default and remove extra role (#9662, @MrFreezeex)
- Terraform gcp can now have extra ingress firewall rules, using new variable
extra_ingress_firewalls
(#9658, @sathieu) - kubeadm/etcd: use config to download certificate (#9609, @MrFreezeex)
Applications
- [argocd] update argocd to v2.5.5 (#9604, @mzaian)
- Upcloud: Reclaim policy for PV is now delete (#9574, @robinAwallace)
- [Exoscale] Add missing zone input variable (#9495, @ayoubeddafali)
- [MetalLB] Avoid MetalLB speaker image download when MetalLB speaker is disabled (#9248, @unai-ttxu)
- [Openstack] Replace deprecated "template" Terraform provider with supported "cloudinit" Terraform provider (#9536, @inflatador)
- [OpenStack] Updated openstack cloud controller to version
v1.25.3
(#9500, @robinAwallace) - [Openstack] Add bastion_allowed_ports to allow custom security group rules on bastion node (#9336, @bl0m1)
- [Openstack] Upgrade 1.22.0 to 1.23.4 (#9332, @QcFe) (See Notes 1)
- [Openstack] Added override variable, additional server groups and cloudinit config (#9452, @Xartos)
- [cinder-csi-nodeplugin] Remove the pods-cloud-data volume (delete upstream) (#9362, @huangkevin404)
- [vsphere-csi] Add missing defaults for external_vsphere_* variables in the csi_driver/vsphere role (#9664, @rlacko58)
- [hetzner] In config, rename ansible groups to use _ instead of - (#9569, @ym)
- [kube-vip] Minor changes on Kube VIP configuration parameters (and fix wrong properties) (#9414, @woutergd)
- [cert-manager] Upgrade to v1.10.1 (#9512, @rtsp) then v1.11.0 (#9661, @mzaian)
- [helm] upgrade to 3.10.3 (#9605, @mzaian)
- [ingress-nginx] upgrade to 1.5.1 (#9532, @mzaian)
- [vSphere] Removing unneeded terraform dependencie & mark vsphere_password as sensitive (#9672, @sathieu)
Container-Managers
- Optimize cgroups settings for node reserved (using new
kube_reserved
, see docs for more information) (#9209, @shelmingsong) - [Docker] Update docker package to 20.10.20 (partial fix for CVE-2022-39253) (#9410, @floryut)
- [containerd] Add support for 1.6.11 (#9544, @yanggangtony)
- [containerd] Added variables for unpriviledged ports and icmp (#9517, @Xartos)
- [containerd] Allow containerd-common to execute multiple times per play (#9543, @chadswen)
- [containerd] Newly started containers will be limited to 16384 open files. To change this number, set
containerd_base_runtime_spec_rlimit_nofile
, or removebase_runtime_spec
from runc runtime to revert to previous behaviour. (#9319, @fungusakafungus) - [containerd] Support v1.6.13 and v1.6.14 (#9585, @yanggangtony)
- [containerd] Add
config_path
var in config.toml.j2 file (#9566, @lengrongfu) - [containerd] Add hashes for containerd versions 1.5.14 , 1.5.15 , 1.5.16 (#9678, @yanggangtony)
- [cri-o] Use cri-o from upstream instead of kubic/OBS (#9374, @cristicalin)
- [nerdctl] upgrade to version 1.0.0 (#9424, @mzaian)
Network
- Bump cni-plugins version to v1.2.0 (#9671, @cyclinder)
- Fix remove Cilium CNI failed because the CNI bin dependency (#9563, @yankay)
- [Calico] Add cni bin when installing (#9367, @ErikJiang)
- [Calico] Add retry for start calico kube controller (#9450, @cleverhu)
- [Calico] Adjust calico-kube-controller pod to non hostNetwork pod (#9465, @cyclinder)
- [Calico] Adjust calico-kube-controller pod to use hostnetwork if using etcd (#9573, @JSpon)
- [Calico] Disable 'Check that IP range is enough for the nodes' (#9491, @mzaian)
- [Calico] Update the tag image to support multiple architectures with the same tag (#9529, @ErikJiang)
- [Calico] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9395, @yankay)
- |Calico] Allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (using
calico_felix_mtu_iface_pattern
) (#9330, @shelmingsong) - [Calico] Replace node-role.kubernetes.io/master with control-plane (#9627, @my-git9)
- [Calico] upgrade default calico version to v3.24.5 (#9580, @yankay)
- [Calico] Add vxlan-v6.calico to the list of NetworkManager unmanaged interfaces (#9631, @cyclinder)
- [Calico] Add retry to avoid 'unknown' state for calicoctl (#9633, @tu1h)
- [Calico] Update Calico VXLAN offload docs because Calico changed the default value (#9639, @yankay)
- [Calico] Add possibility to enable calico floatingIPs feature (using
calico_felix_floating_ips
) (#9680, @MatthieuFin) - [Cilium] Add download configuration for cilium hubble images (using
cilium_enable_hubble
variable) (#9376, @ErikJiang) - [Cilium] Add switch cilium_enable_bandwidth_manager (#9441, @dcwbq)
- [Cilium] Cleanup cilium-init image from cilium template (#9508, @ErikJiang)
- [Cilium] update cilium cli offline download url example (#9458, @cleverhu)
- [Cilium] Install Cilium CLI alongside Cilium (#9436, @dcwbq)
- [flannel] Initcontainer image now correctly support architecture suffix (#9461, @rollandf)
- [flannel] Upgrade version to v0.20.1 (#9528, @ErikJiang)
- [flannel] remove deprecated PodSecurityPolicy (removed in Kubernetes in v1.25) (#9365, @yankay)
- [flannel] Add wireguard encryption backend as option (#9583, @janaurka)
- [flannel] Support dual stack IPv4 & IPv6 networking (#9564, @styshoo)
- [flannel] Allow setting the DirectRouting option on VXLAN (#9438, @willtrnr)
- [flannel] update to v0.20.2 & make it default (#9675, @mzaian)
- [kube-ovn] Update version to v1.10.7 (#9527, @liupeng0518)
- [kube-ovn] Remove kube-ovn log directories when reseting (#9625, @JochenFriedrich)
- [kube-ovn] Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (#9594, @JochenFriedrich)
- [kube-ovn] Support OVN Interconnect (#9599, @JochenFriedrich)
- [multus] added support for mixed type of container engine (#9224, @mr-yaky)
Bug or Regression
- Change
include
toimport_playbook
in recover_control_plane playbook, to support ansible 2.12+ (#9576, @floryut) - Corrected vsphere directory in docs (#9534, @wojciehm)
- Deleting worker nodes is now skipped if there is no
kube_control_plane
node. (#9430, @kerryeon) - Etcd arch can now support arm64 and amd64 (#9421, @yanggangtony)
- Fix cert-manager deployment on hardening environments (#9404, @oomichi)
- Fix checksum of ciliumcli v0.12.5 for arm64 (#9614, @oomichi)
- Fix inconsistent handling of admission plugin list (
kube_apiserver_enable_admission_plugins
must be ...