{"id":345,"date":"2024-09-24T20:25:10","date_gmt":"2024-09-24T18:25:10","guid":{"rendered":"https:\/\/www.c-sautter.de\/?p=345"},"modified":"2024-11-23T15:18:39","modified_gmt":"2024-11-23T14:18:39","slug":"gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte","status":"publish","type":"post","link":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","title":{"rendered":"GitLab CI\/CD vs. Heat: How a Hardware Failure Sabotaged CI\/CD Jobs"},"content":{"rendered":"
\"Gitlab<\/figure>\n\n\n

A ROCK 5 A<\/a> has become a node in my Homelab Kubernetes (k8s) cluster. For example, Gitlab CI jobs with container builds (Buildah, Kaniko) are executed on it. An unusually high error rate in the CI\/CD system turned out to be a hardware error after some debugging and extensive troubleshooting. After several years of experience with Gitlab CI, this was one of the \u201ccraziest\u201d bugs for me.<\/p>\n\n\n\n

Error description<\/h2>\n\n\n\n

The Gitlab CI jobs are hosted in a Kubernetes cluster via the Gitlab Runner Kubernetes executor<\/a>. For native arm64 builds, a ROCK 5 A<\/a> is used as a node in the cluster.<\/p>\n\n\n\n

From the beginning, my CI jobs were experiencing sporadic errors - TLS connections, layer caching, compression, decompression, and container pull\/push. Due to the test status and continuous optimization of various settings in the CI jobs and the Gitlab Runner, I assumed that the sporadic errors could be caused by misconfigurations or insufficient resources, such as insufficient memory.<\/p>\n\n\n\n

As part of further stabilization measures, I therefore checked and adjusted the resource requests and limits<\/a> in particular. There were always phases in which everything worked smoothly until various errors suddenly occurred again.<\/p>\n\n\n\n

Error messages related to TLS\/SSL or HTTPS connections also suggested that there might be a problem with the network or internet connection. However, an initial analysis of the traffic with Wireshark showed nothing unusual. Since most of the errors only occurred on the arm64 node, I suspected a problem related to the arm64 CPU architecture.<\/p>\n\n\n\n

Only a parallel test with different build tools such as Buildah and Skopeo produced noticeable results \u2013 both tools produced almost identical sporadic error patterns.<\/p>\n\n\n\n

Various error patterns in detail<\/h2>\n\n\n\n

tls: bad record MAC<\/h3>\n\n\n\n
tls: bad record MAC<\/span><\/path><\/path><\/svg><\/span>
$ \/syft scan ${TRACE+-vv} $DOCKER_SNAPSHOT_IMAGE $DOCKER_SBOM_OPTS -o cyclonedx-json=reports\/docker-sbom-${basename}.cyclonedx.json<\/span><\/span>\ncould not determine source: errors occurred attempting to resolve 'registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-feat-retry-build-amd64':<\/span><\/span>\n  - docker: docker not available: failed to connect to Docker daemon. Ensure Docker is running and accessible<\/span><\/span>\n  - podman: podman not available: no host address<\/span><\/span>\n  - containerd: containerd not available: no grpc connection or services is available: unavailable<\/span><\/span>\n  - oci-registry: unable to populate layer cache dir="\/tmp\/stereoscope-3483325947\/oci-registry-image-1395683340\/sha256:d7d53246cf32b5617c62ec6db76daadad242335800b3767fb8ab5372d0c24df7" : local error: tls: bad record MAC<\/span><\/span>\n  - additionally, the following providers failed with file does not exist: docker-archive, oci-archive, oci-dir, singularity, oci-dir, local-file, local-directory<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
tls: bad record MAC<\/span><\/path><\/path><\/svg><\/span>
51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.59 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.59 MiB p\/s ETA 0s<\/span><\/span>\n51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.49 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.49 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [---------------------------------------------------] 100.00% 1.92 MiB p\/s 27s<\/span><\/span>\n2024\/08\/17 16:30:30 WARN '--vuln-type' is deprecated. Use '--pkg-types' instead.<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[vuln] Vulnerability scanning is enabled<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] Secret scanning is enabled<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] Please see also https:\/\/aquasecurity.github.io\/trivy\/v0.54\/docs\/scanner\/secret#recommendation for faster secret detection<\/span><\/span>\n2024-08-17T16:31:53Z\tFATAL\tFatal error\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4b6bc5885679abe54fc17878bb74d45875f6b7747e9ff89adf0620a631354146): walk error: failed to extract the archive: local error: tls: bad record MAC<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
error verifying sha256 checksum after reading x bytes; got \u201csha256:x\u201d, want \u201csha256:y\u201d<\/h3>\n\n\n\n
error verifying sha256 checksum after reading x bytes; got „sha256:x“, want „sha256:y“<\/span><\/path><\/path><\/svg><\/span>
$ \/syft scan ${TRACE+-vv} $DOCKER_SNAPSHOT_IMAGE $DOCKER_SBOM_OPTS -o cyclonedx-json=reports\/docker-sbom-${basename}.cyclonedx.json<\/span><\/span>\ncould not determine source: errors occurred attempting to resolve 'registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-main-amd64':<\/span><\/span>\n  - docker: docker not available: failed to connect to Docker daemon. Ensure Docker is running and accessible<\/span><\/span>\n  - podman: podman not available: no host address<\/span><\/span>\n  - containerd: containerd not available: no grpc connection or services is available: unavailable<\/span><\/span>\n  - oci-registry: unable to populate layer cache dir="\/tmp\/stereoscope-601115183\/oci-registry-image-1244415430\/sha256:cc9e89b59b07f31c85950499be0fb00281e767f543e3518240cadef521456b0d" : error verifying sha256 checksum after reading 48836061 bytes; got "sha256:e615f70755ea5f5e51525e97c94f7f8eb79df2f61427e5cb549676fae6e3a19e", want "sha256:8314597ff06fa934871480701d61ccf24999f5a025e70a7a196e5b1f9690cae4"<\/span><\/span>\n  - additionally, the following providers failed with file does not exist: docker-archive, oci-archive, oci-dir, singularity, oci-dir, local-file, local-directory<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
error building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/h3>\n\n\n\n
error building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/span><\/path><\/path><\/svg><\/span>
INFO[0018] Retrieving image manifest ghcr.io\/containerd\/nerdctl:v1.7.6@sha256:c7ed7c98f3da0d60d1b0b3e897460bd6fe306a7e4febe600533da035a3651f0e <\/span><\/span>\nINFO[0018] Returning cached image manifest              <\/span><\/span>\nINFO[0018] Executing 0 build triggers                   <\/span><\/span>\nINFO[0018] Building stage 'ghcr.io\/containerd\/nerdctl:v1.7.6@sha256:c7ed7c98f3da0d60d1b0b3e897460bd6fe306a7e4febe600533da035a3651f0e' [idx: '2', base-idx: '-1'] <\/span><\/span>\nerror building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:x Reason:map[]]<\/h3>\n\n\n\n
DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:x Reason:map[]]<\/span><\/path><\/path><\/svg><\/span>
INFO[0446] Pushing image to registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-renovate-ghcr-io-terraform-linters-tflint-0-x-arm64 <\/span><\/span>\nerror pushing image: failed to push to destination registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-renovate-ghcr-io-terraform-linters-tflint-0-x-arm64: PUT https:\/\/registry.gitlab.com\/v2\/csautter\/gitlab-ci-cd-templates\/snapshot\/blobs\/uploads\/604cd097-2ce3-4ae9-8744-30b39ce78849?_state=REDACTED&digest=sha256%3A2348448717b77b6e478a55d3d317a57eb4d6630f09bf36b038946749daa893a1: DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:2348448717b77b6e478a55d3d317a57eb4d6630f09bf36b038946749daa893a1 Reason:map[]]<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got \u201csha256:x\u201d, want \u201csha256:y\u201d<\/h3>\n\n\n\n
Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got „sha256:x“, want „sha256:y“<\/span><\/path><\/path><\/svg><\/span>
49.91 MiB \/ 51.51 MiB [----------------------------------------------->_] 96.88% 1.79 MiB p\/s ETA 0s50.39 MiB \/ 51.51 MiB [----------------------------------------------->_] 97.82% 1.79 MiB p\/s ETA 0s<\/span><\/span>\n50.94 MiB \/ 51.51 MiB [------------------------------------------------>] 98.88% 1.79 MiB p\/s ETA 0s51.45 MiB \/ 51.51 MiB [------------------------------------------------>] 99.88% 1.84 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [---------------------------------------------------] 100.00% 1.99 MiB p\/s 26s<\/span><\/span>\n2024-08-17T16:30:30Z\tFATAL\tFatal error\tinit error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got "sha256:48972fb93363b0612c2bc4f715a82b92ed8387bbfffeae77dbe7f557918130c3", want "sha256:f0bef38f54917f45165ae49dfcbbbeddf0d02bb9f698946390be766a0ee3f8cf"<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
System test<\/h2>\n\n\n\n
To identify hardware problems, I tested the CPU with stress-ng<\/code> in verify mode. Some cores showed clear errors.<\/p>\n\n\n\n
Stress test the CPU with stress-ng in verify mode<\/h3>\n\n\n\n
Numerous calculation errors occurred, such as inaccurate calculations of constants or incorrect square roots.<\/p>\n\n\n\n
test with stress-ng<\/span><\/path><\/path><\/svg><\/span>
$ sudo apt-get install stress-ng<\/span><\/span>\n$ stress-ng --cpu 0 --verify --verbose --timeout 5m<\/span><\/span>\nstress-ng: debug: [886181] stress-ng 0.13.12<\/span><\/span>\nstress-ng: debug: [886181] system: Linux rock-5a 5.10.110-37-rockchip #27a257394 SMP Thu May 23 02:38:59 UTC 2024 aarch64<\/span><\/span>\nstress-ng: debug: [886181] RAM total: 15.6G, RAM free: 4.1G, swap free: 7.8G<\/span><\/span>\nstress-ng: debug: [886181] 8 processors online, 8 processors configured<\/span><\/span>\nstress-ng: info:  [886181] setting to a 300 second (5 mins, 0.00 secs) run per stressor<\/span><\/span>\nstress-ng: info:  [886181] dispatching hogs: 8 cpu<\/span><\/span>\nstress-ng: debug: [886181] cache allocate: shared cache buffer size: 3072K<\/span><\/span>\nstress-ng: debug: [886181] starting stressors<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu: started [886182] (instance 0)<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu: started [886183] (instance 1)<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu: started [886184] (instance 2)<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu: started [886185] (instance 3)<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu: started [886186] (instance 4)<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu: started [886187] (instance 5)<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu: started [886188] (instance 6)<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886181] 8 stressors started<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu: started [886189] (instance 7)<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: sqrtf error detected on sqrt(2308358344)<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886186] stress-ng-cpu: Ap\u00e9ry's const not accurate enough<\/span><\/span>\nstress-ng: fail:  [886186] stress-ng-cpu: sqrtf error detected on sqrt(3853404901)<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: Ap\u00e9ry's const not accurate enough<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Stirling's approximation of factorial(59) out of range<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: sqrtf error detected on sqrt(1186671297)<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: sqrtf error detected on sqrt(2136759488)<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Newton-Raphson sqrt computation took more iterations than expected<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Newton-Raphson sqrt not accurate enough<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: sqrtf error detected on sqrt(1951583473)<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: Stirling's approximation of factorial(117) out of range<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886184] stress-ng-cpu: prime error detected, number of primes has been miscalculated<\/span><\/span>\nstress-ng: fail:  [886184] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886189] stress-ng-cpu: sqrtf error detected on sqrt(3283974744)<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: sqrtf error detected on sqrt(2730736404)<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886189] stress-ng-cpu: sqrtf error detected on sqrt(4136192236)<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: Stirling's approximation of factorial(100) out of range<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\ninfo: 5 failures reached, aborting stress process<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu: exited [886183] (instance 1)<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu: exited [886185] (instance 3)<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu: exited [886186] (instance 4)<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu: exited [886182] (instance 0)<\/span><\/span>\nstress-ng: debug: [886181] process [886182] terminated<\/span><\/span>\nstress-ng: error: [886181] process 886183 (stress-ng-cpu) terminated with an error, exit status=2 (stressor failed)<\/span><\/span>\nstress-ng: debug: [886181] process [886183] terminated<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu: exited [886187] (instance 5)<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu: exited [886188] (instance 6)<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu: exited [886184] (instance 2)<\/span><\/span>\nstress-ng: debug: [886181] process [886184] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886185] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886186] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886187] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886188] terminated<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu: exited [886189] (instance 7)<\/span><\/span>\nstress-ng: debug: [886181] process [886189] terminated<\/span><\/span>\nstress-ng: info:  [886181] unsuccessful run completed in 300.10s (5 mins, 0.10 secs)<\/span><\/span>\nstress-ng: debug: [886181] metrics-check: all stressor metrics validated and sane<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
As a workaround, I temporarily deactivated the faulty cores. This temporarily prevented the errors in the CI jobs.<\/p>\n\n\n\n
disable defective cpu core<\/span><\/path><\/path><\/svg><\/span>
cd<\/span> <\/span>\/sys\/devices\/system\/cpu<\/span><\/span>\nls<\/span> <\/span>cpu<\/span>*<\/span><\/span>\n# disable specific core<\/span><\/span>\necho<\/span> <\/span>0<\/span> <\/span>><\/span> <\/span>\/sys\/devices\/system\/cpu\/cpu3\/online<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Later, errors occurred again despite the cores being deactivated.<\/p>\n\n\n\n
Monitoring CPU temperature with lm-sensors<\/h3>\n\n\n\n
In the next step, I reactivated all cores and monitored the CPU temperature during the stress test using the lm-sensors<\/code> package.<\/p>\n\n\n\n
watch temperatures with lm-sensors<\/span><\/path><\/path><\/svg><\/span>
$ sudo apt-get install lm-sensors<\/span><\/span>\n$ sudo sensors-detect<\/span><\/span>\n$ sudo watch sensors<\/span><\/span>\n<\/span><\/span>\nEvery 2.0s: sensors                                                                    rock-5a: Tue Aug 20 19:48:13 2024<\/span><\/span>\n<\/span><\/span>\nnpu_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +72.1\u00b0C<\/span><\/span>\n<\/span><\/span>\ncenter_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +70.2\u00b0C<\/span><\/span>\n<\/span><\/span>\nbigcore1_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +79.5\u00b0C<\/span><\/span>\n<\/span><\/span>\nsoc_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +75.8\u00b0C  (crit = +115.0\u00b0C)<\/span><\/span>\n<\/span><\/span>\ngpu_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +69.3\u00b0C<\/span><\/span>\n<\/span><\/span>\nlittlecore_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +76.7\u00b0C<\/span><\/span>\n<\/span><\/span>\nbigcore0_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +79.5\u00b0C<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
The CPU temperature rose to over 80\u00b0C in some cases, which is theoretically still within the permissible range. However, I could observe an increase in faulty calculations at temperatures of around 70-80\u00b0C.<\/p>\n\n\n\n
\n
The SoC (RK3588S) is specified to limit its maximum internal temperature to 80\u00b0C before
throttling the clock speeds to maintain reliability within the allowed temperature range. If
the ROCK 5A is intended to be used continuously in high performance applications, it may
be necessary to use external cooling methods (for example, heat sink, fan, etc.) which will
allow the SoC to continue running at maximum clock speed indefinitely below its prede\u2011
fined 80\u00b0C peak temperature limiter.<\/p>\n<\/blockquote>\n\n\n\n
Source: https:\/\/docs.rs-online.com\/a9f0\/A700000010117420.pdf<\/a><\/p>\n\n\n\n
A further stress test with an additional external fan reduced the SoC temperature to below 50\u00b0C and the errors disappeared completely.<\/p>\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Apparently my ROCK 5 A<\/a> works within the specified temperature specification, but additional cooling is required to ensure error-free operation. The error was particularly interesting because at first glance it did not indicate a hardware problem and the resulting error messages were very unspecific.<\/p>\n\n\n\n
I reported the error to the dealer RS \u200b\u200bComponents and received a replacement device for my defective ROCK 5 A<\/a> without any problems. With the new ROCK 5 A I no longer have any problems.<\/p>","protected":false},"excerpt":{"rendered":"
A ROCK 5 A has recently been running as a node in my home lab Kubernetes (k8s) cluster. Gitlab CI jobs with container builds (Buildah, Kaniko) are run on it, for example. After some debugging and extensive troubleshooting, an unusually high error rate in the CI\/CD system turned out to be a hardware error. After several years of experience with Gitlab CI, this was...<\/p>\n
Read More<\/a><\/p>","protected":false},"author":1,"featured_media":446,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43,42,39],"tags":[48,47,49,34],"class_list":["post-345","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-build-automation","category-continuous-deployment","category-continuous-integration","tag-cd","tag-ci","tag-ci-cd","tag-gitlab"],"yoast_head":"\nGitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter<\/title>\n<meta name=\"description\" content=\"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter\" \/>\n<meta property=\"og:description\" content=\"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\" \/>\n<meta property=\"og:site_name\" content=\"Carl-Christian Sautter\" \/>\n<meta property=\"article:published_time\" content=\"2024-09-24T18:25:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-11-23T14:18:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1080\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"chris\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"chris\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\"},\"author\":{\"name\":\"chris\",\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d\"},\"headline\":\"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte\",\"datePublished\":\"2024-09-24T18:25:10+00:00\",\"dateModified\":\"2024-11-23T14:18:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\"},\"wordCount\":782,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d\"},\"image\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\",\"keywords\":[\"cd\",\"ci\",\"CI\/CD\",\"gitlab\"],\"articleSection\":[\"Build Automation\",\"Continuous Deployment\",\"Continuous Integration\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\",\"url\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\",\"name\":\"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter\",\"isPartOf\":{\"@id\":\"https:\/\/www.c-sautter.de\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\",\"datePublished\":\"2024-09-24T18:25:10+00:00\",\"dateModified\":\"2024-11-23T14:18:39+00:00\",\"description\":\"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage\",\"url\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\",\"contentUrl\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\",\"width\":1080,\"height\":1080,\"caption\":\"Gitlab CI\/CD nodes overheat\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.c-sautter.de\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.c-sautter.de\/#website\",\"url\":\"https:\/\/www.c-sautter.de\/\",\"name\":\"Carl-Christian Sautter\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.c-sautter.de\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d\",\"name\":\"chris\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg\",\"contentUrl\":\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg\",\"width\":1657,\"height\":1657,\"caption\":\"chris\"},\"logo\":{\"@id\":\"https:\/\/www.c-sautter.de\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/www.c-sautter.de\",\"https:\/\/www.linkedin.com\/in\/carl-christian-sautter\/\"],\"url\":\"https:\/\/www.c-sautter.de\/en\/author\/chris\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","og_locale":"en_US","og_type":"article","og_title":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","og_description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","og_url":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","og_site_name":"Carl-Christian Sautter","article_published_time":"2024-09-24T18:25:10+00:00","article_modified_time":"2024-11-23T14:18:39+00:00","og_image":[{"width":1080,"height":1080,"url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","type":"image\/jpeg"}],"author":"chris","twitter_card":"summary_large_image","twitter_misc":{"Written by":"chris","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#article","isPartOf":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"},"author":{"name":"chris","@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"headline":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte","datePublished":"2024-09-24T18:25:10+00:00","dateModified":"2024-11-23T14:18:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"},"wordCount":782,"commentCount":0,"publisher":{"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"image":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"thumbnailUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","keywords":["cd","ci","CI\/CD","gitlab"],"articleSection":["Build Automation","Continuous Deployment","Continuous Integration"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","url":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","name":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","isPartOf":{"@id":"https:\/\/www.c-sautter.de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"image":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"thumbnailUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","datePublished":"2024-09-24T18:25:10+00:00","dateModified":"2024-11-23T14:18:39+00:00","description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","breadcrumb":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage","url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","contentUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","width":1080,"height":1080,"caption":"Gitlab CI\/CD nodes overheat"},{"@type":"BreadcrumbList","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.c-sautter.de\/"},{"@type":"ListItem","position":2,"name":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte"}]},{"@type":"WebSite","@id":"https:\/\/www.c-sautter.de\/#website","url":"https:\/\/www.c-sautter.de\/","name":"Carl-Christian Sautter","description":"","publisher":{"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.c-sautter.de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d","name":"chris","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/image\/","url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg","contentUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg","width":1657,"height":1657,"caption":"chris"},"logo":{"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/www.c-sautter.de","https:\/\/www.linkedin.com\/in\/carl-christian-sautter\/"],"url":"https:\/\/www.c-sautter.de\/en\/author\/chris\/"}]}},"_links":{"self":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/comments?post=345"}],"version-history":[{"count":19,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345\/revisions"}],"predecessor-version":[{"id":448,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345\/revisions\/448"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/media\/446"}],"wp:attachment":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/media?parent=345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/categories?post=345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/tags?post=345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}