{"id":345,"date":"2024-09-24T20:25:10","date_gmt":"2024-09-24T18:25:10","guid":{"rendered":"https:\/\/www.c-sautter.de\/?p=345"},"modified":"2024-11-23T15:18:39","modified_gmt":"2024-11-23T14:18:39","slug":"gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte","status":"publish","type":"post","link":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","title":{"rendered":"GitLab CI\/CD vs. Heat: How a Hardware Failure Sabotaged CI\/CD Jobs"},"content":{"rendered":"
\"Gitlab<\/figure>\n\n\n

In meinem Homelab Kubernetes (k8s) Cluster l\u00e4uft seit kurzem ein ROCK 5 A<\/a> als Node. Darauf werden beispielsweise Gitlab CI Jobs mit Container Builds (Buildah, Kaniko) ausgef\u00fchrt. Eine ungew\u00f6hnlich hohe Fehlerrate im CI\/CD System hat sich nach einigem Debugging und umfassender Fehlersuche als Hardware Fehler entpuppt. Nach mehreren Jahren Erfahrung mit Gitlab CI war das f\u00fcr mich einer der „verr\u00fccktesten“ Bugs.<\/p>\n\n\n\n

Fehlerbeschreibung<\/h2>\n\n\n\n

Die Gitlab CI Jobs werden in einem Kubernetes Cluster \u00fcber den Gitlab Runner Kubernetes executor<\/a> ausgef\u00fchrt. F\u00fcr native arm64-Builds dient unter anderem ein ROCK 5 A<\/a> als Node im Cluster.<\/p>\n\n\n\n

Von Anfang an traten in meinen CI Jobs sporadische Fehler auf \u2013 bei TLS-Verbindungen, Layer-Caching, Kompression, Dekompression sowie beim Container-Pull\/Push. Aufgrund des Teststatus und der kontinuierlichen Optimierung verschiedener Einstellungen in den CI Jobs und im Gitlab Runner nahm ich an, dass die sporadischen Fehler durch Fehlkonfigurationen oder unzureichende Ressourcen, wie zu wenig Arbeitsspeicher, verursacht werden k\u00f6nnten.<\/p>\n\n\n\n

Im Rahmen weiterer Stabilisierungsma\u00dfnahmen habe ich daher insbesondere die Ressourcen Requests und Limits<\/a> \u00fcberpr\u00fcft und angepasst. Immer wieder gab es Phasen, in denen alles reibungslos funktionierte, bis pl\u00f6tzlich erneut verschiedene Fehler auftraten.<\/p>\n\n\n\n

Auch Fehlermeldungen im Zusammenhang mit TLS\/SSL bzw. HTTPS-Verbindungen lie\u00dfen vermuten, dass es ein Problem mit der Netzwerk- oder Internetverbindung geben k\u00f6nnte. Eine erste Analyse des Traffics mit Wireshark zeigte jedoch keine Auff\u00e4lligkeiten. Da die meisten Fehler nur auf dem arm64-Node auftraten, vermutete ich ein Problem im Zusammenhang mit der arm64-CPU-Architektur.<\/p>\n\n\n\n

Erst ein paralleler Test mit verschiedenen Build-Tools wie Buildah und Skopeo brachte auff\u00e4llige Ergebnisse \u2013 beide Tools erzeugten fast identische sporadische Fehlerbilder.<\/p>\n\n\n\n

Verschiedene Fehlerbilder im Detail<\/h2>\n\n\n\n

tls: bad record MAC<\/h3>\n\n\n\n
tls: bad record MAC<\/span><\/path><\/path><\/svg><\/span>
$ \/syft scan ${TRACE+-vv} $DOCKER_SNAPSHOT_IMAGE $DOCKER_SBOM_OPTS -o cyclonedx-json=reports\/docker-sbom-${basename}.cyclonedx.json<\/span><\/span>\ncould not determine source: errors occurred attempting to resolve 'registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-feat-retry-build-amd64':<\/span><\/span>\n  - docker: docker not available: failed to connect to Docker daemon. Ensure Docker is running and accessible<\/span><\/span>\n  - podman: podman not available: no host address<\/span><\/span>\n  - containerd: containerd not available: no grpc connection or services is available: unavailable<\/span><\/span>\n  - oci-registry: unable to populate layer cache dir="\/tmp\/stereoscope-3483325947\/oci-registry-image-1395683340\/sha256:d7d53246cf32b5617c62ec6db76daadad242335800b3767fb8ab5372d0c24df7" : local error: tls: bad record MAC<\/span><\/span>\n  - additionally, the following providers failed with file does not exist: docker-archive, oci-archive, oci-dir, singularity, oci-dir, local-file, local-directory<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
tls: bad record MAC<\/span><\/path><\/path><\/svg><\/span>
51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.59 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.59 MiB p\/s ETA 0s<\/span><\/span>\n51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.49 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [----------------------------------------------->] 100.00% 1.49 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [---------------------------------------------------] 100.00% 1.92 MiB p\/s 27s<\/span><\/span>\n2024\/08\/17 16:30:30 WARN '--vuln-type' is deprecated. Use '--pkg-types' instead.<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[vuln] Vulnerability scanning is enabled<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] Secret scanning is enabled<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning<\/span><\/span>\n2024-08-17T16:30:30Z\tINFO\t[secret] Please see also https:\/\/aquasecurity.github.io\/trivy\/v0.54\/docs\/scanner\/secret#recommendation for faster secret detection<\/span><\/span>\n2024-08-17T16:31:53Z\tFATAL\tFatal error\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:4b6bc5885679abe54fc17878bb74d45875f6b7747e9ff89adf0620a631354146): walk error: failed to extract the archive: local error: tls: bad record MAC<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
error verifying sha256 checksum after reading x bytes; got „sha256:x“, want „sha256:y“<\/h3>\n\n\n\n
error verifying sha256 checksum after reading x bytes; got „sha256:x“, want „sha256:y“<\/span><\/path><\/path><\/svg><\/span>
$ \/syft scan ${TRACE+-vv} $DOCKER_SNAPSHOT_IMAGE $DOCKER_SBOM_OPTS -o cyclonedx-json=reports\/docker-sbom-${basename}.cyclonedx.json<\/span><\/span>\ncould not determine source: errors occurred attempting to resolve 'registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-main-amd64':<\/span><\/span>\n  - docker: docker not available: failed to connect to Docker daemon. Ensure Docker is running and accessible<\/span><\/span>\n  - podman: podman not available: no host address<\/span><\/span>\n  - containerd: containerd not available: no grpc connection or services is available: unavailable<\/span><\/span>\n  - oci-registry: unable to populate layer cache dir="\/tmp\/stereoscope-601115183\/oci-registry-image-1244415430\/sha256:cc9e89b59b07f31c85950499be0fb00281e767f543e3518240cadef521456b0d" : error verifying sha256 checksum after reading 48836061 bytes; got "sha256:e615f70755ea5f5e51525e97c94f7f8eb79df2f61427e5cb549676fae6e3a19e", want "sha256:8314597ff06fa934871480701d61ccf24999f5a025e70a7a196e5b1f9690cae4"<\/span><\/span>\n  - additionally, the following providers failed with file does not exist: docker-archive, oci-archive, oci-dir, singularity, oci-dir, local-file, local-directory<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
error building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/h3>\n\n\n\n
error building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/span><\/path><\/path><\/svg><\/span>
INFO[0018] Retrieving image manifest ghcr.io\/containerd\/nerdctl:v1.7.6@sha256:c7ed7c98f3da0d60d1b0b3e897460bd6fe306a7e4febe600533da035a3651f0e <\/span><\/span>\nINFO[0018] Returning cached image manifest              <\/span><\/span>\nINFO[0018] Executing 0 build triggers                   <\/span><\/span>\nINFO[0018] Building stage 'ghcr.io\/containerd\/nerdctl:v1.7.6@sha256:c7ed7c98f3da0d60d1b0b3e897460bd6fe306a7e4febe600533da035a3651f0e' [idx: '2', base-idx: '-1'] <\/span><\/span>\nerror building image: error building stage: failed to get filesystem from image: error reading tar 3: archive\/tar: invalid tar header<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:x Reason:map[]]<\/h3>\n\n\n\n
DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:x Reason:map[]]<\/span><\/path><\/path><\/svg><\/span>
INFO[0446] Pushing image to registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-renovate-ghcr-io-terraform-linters-tflint-0-x-arm64 <\/span><\/span>\nerror pushing image: failed to push to destination registry.gitlab.com\/csautter\/gitlab-ci-cd-templates\/snapshot:3tcd-renovate-ghcr-io-terraform-linters-tflint-0-x-arm64: PUT https:\/\/registry.gitlab.com\/v2\/csautter\/gitlab-ci-cd-templates\/snapshot\/blobs\/uploads\/604cd097-2ce3-4ae9-8744-30b39ce78849?_state=REDACTED&digest=sha256%3A2348448717b77b6e478a55d3d317a57eb4d6630f09bf36b038946749daa893a1: DIGEST_INVALID: provided digest did not match uploaded content; map[Digest:sha256:2348448717b77b6e478a55d3d317a57eb4d6630f09bf36b038946749daa893a1 Reason:map[]]<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got „sha256:x“, want „sha256:y“<\/h3>\n\n\n\n
Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got „sha256:x“, want „sha256:y“<\/span><\/path><\/path><\/svg><\/span>
49.91 MiB \/ 51.51 MiB [----------------------------------------------->_] 96.88% 1.79 MiB p\/s ETA 0s50.39 MiB \/ 51.51 MiB [----------------------------------------------->_] 97.82% 1.79 MiB p\/s ETA 0s<\/span><\/span>\n50.94 MiB \/ 51.51 MiB [------------------------------------------------>] 98.88% 1.79 MiB p\/s ETA 0s51.45 MiB \/ 51.51 MiB [------------------------------------------------>] 99.88% 1.84 MiB p\/s ETA 0s51.51 MiB \/ 51.51 MiB [---------------------------------------------------] 100.00% 1.99 MiB p\/s 26s<\/span><\/span>\n2024-08-17T16:30:30Z\tFATAL\tFatal error\tinit error: DB error: failed to download vulnerability DB: database download error: oci download error: copy error: error verifying sha256 checksum after reading 54016433 bytes; got "sha256:48972fb93363b0612c2bc4f715a82b92ed8387bbfffeae77dbe7f557918130c3", want "sha256:f0bef38f54917f45165ae49dfcbbbeddf0d02bb9f698946390be766a0ee3f8cf"<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Systemtest<\/h2>\n\n\n\n
Um Hardwareprobleme zu identifizieren, habe ich die CPU mit stress-ng<\/code> im Verify-Mode getestet. Dabei zeigten einige Cores deutliche Fehler.<\/p>\n\n\n\n
Stresstest der CPU mit stress-ng im verify mode<\/h3>\n\n\n\n
Es traten zahlreiche Rechenfehler auf, wie ungenaue Berechnungen von Konstanten oder fehlerhafte Quadratwurzeln.<\/p>\n\n\n\n
test with stress-ng<\/span><\/path><\/path><\/svg><\/span>
$ sudo apt-get install stress-ng<\/span><\/span>\n$ stress-ng --cpu 0 --verify --verbose --timeout 5m<\/span><\/span>\nstress-ng: debug: [886181] stress-ng 0.13.12<\/span><\/span>\nstress-ng: debug: [886181] system: Linux rock-5a 5.10.110-37-rockchip #27a257394 SMP Thu May 23 02:38:59 UTC 2024 aarch64<\/span><\/span>\nstress-ng: debug: [886181] RAM total: 15.6G, RAM free: 4.1G, swap free: 7.8G<\/span><\/span>\nstress-ng: debug: [886181] 8 processors online, 8 processors configured<\/span><\/span>\nstress-ng: info:  [886181] setting to a 300 second (5 mins, 0.00 secs) run per stressor<\/span><\/span>\nstress-ng: info:  [886181] dispatching hogs: 8 cpu<\/span><\/span>\nstress-ng: debug: [886181] cache allocate: shared cache buffer size: 3072K<\/span><\/span>\nstress-ng: debug: [886181] starting stressors<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu: started [886182] (instance 0)<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu: started [886183] (instance 1)<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu: started [886184] (instance 2)<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu: started [886185] (instance 3)<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu: started [886186] (instance 4)<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu: started [886187] (instance 5)<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu: started [886188] (instance 6)<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886181] 8 stressors started<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu: started [886189] (instance 7)<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu using method 'all'<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: sqrtf error detected on sqrt(2308358344)<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886186] stress-ng-cpu: Ap\u00e9ry's const not accurate enough<\/span><\/span>\nstress-ng: fail:  [886186] stress-ng-cpu: sqrtf error detected on sqrt(3853404901)<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: Ap\u00e9ry's const not accurate enough<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Stirling's approximation of factorial(59) out of range<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: sqrtf error detected on sqrt(1186671297)<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: sqrtf error detected on sqrt(2136759488)<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Newton-Raphson sqrt computation took more iterations than expected<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: Newton-Raphson sqrt not accurate enough<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: sqrtf error detected on sqrt(1951583473)<\/span><\/span>\nstress-ng: fail:  [886187] stress-ng-cpu: Stirling's approximation of factorial(117) out of range<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886182] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886184] stress-ng-cpu: prime error detected, number of primes has been miscalculated<\/span><\/span>\nstress-ng: fail:  [886184] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886189] stress-ng-cpu: sqrtf error detected on sqrt(3283974744)<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: sqrtf error detected on sqrt(2730736404)<\/span><\/span>\nstress-ng: fail:  [886188] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\nstress-ng: fail:  [886189] stress-ng-cpu: sqrtf error detected on sqrt(4136192236)<\/span><\/span>\nstress-ng: fail:  [886185] stress-ng-cpu: Stirling's approximation of factorial(100) out of range<\/span><\/span>\nstress-ng: fail:  [886183] stress-ng-cpu: calculation of Euler-Mascheroni constant not as accurate as expected<\/span><\/span>\ninfo: 5 failures reached, aborting stress process<\/span><\/span>\nstress-ng: debug: [886183] stress-ng-cpu: exited [886183] (instance 1)<\/span><\/span>\nstress-ng: debug: [886185] stress-ng-cpu: exited [886185] (instance 3)<\/span><\/span>\nstress-ng: debug: [886186] stress-ng-cpu: exited [886186] (instance 4)<\/span><\/span>\nstress-ng: debug: [886182] stress-ng-cpu: exited [886182] (instance 0)<\/span><\/span>\nstress-ng: debug: [886181] process [886182] terminated<\/span><\/span>\nstress-ng: error: [886181] process 886183 (stress-ng-cpu) terminated with an error, exit status=2 (stressor failed)<\/span><\/span>\nstress-ng: debug: [886181] process [886183] terminated<\/span><\/span>\nstress-ng: debug: [886187] stress-ng-cpu: exited [886187] (instance 5)<\/span><\/span>\nstress-ng: debug: [886188] stress-ng-cpu: exited [886188] (instance 6)<\/span><\/span>\nstress-ng: debug: [886184] stress-ng-cpu: exited [886184] (instance 2)<\/span><\/span>\nstress-ng: debug: [886181] process [886184] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886185] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886186] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886187] terminated<\/span><\/span>\nstress-ng: debug: [886181] process [886188] terminated<\/span><\/span>\nstress-ng: debug: [886189] stress-ng-cpu: exited [886189] (instance 7)<\/span><\/span>\nstress-ng: debug: [886181] process [886189] terminated<\/span><\/span>\nstress-ng: info:  [886181] unsuccessful run completed in 300.10s (5 mins, 0.10 secs)<\/span><\/span>\nstress-ng: debug: [886181] metrics-check: all stressor metrics validated and sane<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Als Workaround habe ich die fehlerhaften Cores tempor\u00e4r deaktiviert. Zeitweise konnten dadurch die Fehler in den CI Jobs verhindert werden.<\/p>\n\n\n\n
disable defective cpu core<\/span><\/path><\/path><\/svg><\/span>
cd<\/span> <\/span>\/sys\/devices\/system\/cpu<\/span><\/span>\nls<\/span> <\/span>cpu<\/span>*<\/span><\/span>\n# disable specific core<\/span><\/span>\necho<\/span> <\/span>0<\/span> <\/span>><\/span> <\/span>\/sys\/devices\/system\/cpu\/cpu3\/online<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Sp\u00e4ter traten trotz deaktivierter Cores erneut Fehler auf.<\/p>\n\n\n\n
\u00dcberwachen der CPU Temperatur mit lm-sensors<\/h3>\n\n\n\n
Im n\u00e4chsten Schritt habe ich alle Cores wieder aktiviert und w\u00e4hrend des Stresstests die CPU-Temperatur mit dem Paket lm-sensors<\/code> \u00fcberwacht.<\/p>\n\n\n\n
watch temperatures with lm-sensors<\/span><\/path><\/path><\/svg><\/span>
$ sudo apt-get install lm-sensors<\/span><\/span>\n$ sudo sensors-detect<\/span><\/span>\n$ sudo watch sensors<\/span><\/span>\n<\/span><\/span>\nEvery 2.0s: sensors                                                                    rock-5a: Tue Aug 20 19:48:13 2024<\/span><\/span>\n<\/span><\/span>\nnpu_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +72.1\u00b0C<\/span><\/span>\n<\/span><\/span>\ncenter_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +70.2\u00b0C<\/span><\/span>\n<\/span><\/span>\nbigcore1_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +79.5\u00b0C<\/span><\/span>\n<\/span><\/span>\nsoc_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +75.8\u00b0C  (crit = +115.0\u00b0C)<\/span><\/span>\n<\/span><\/span>\ngpu_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +69.3\u00b0C<\/span><\/span>\n<\/span><\/span>\nlittlecore_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +76.7\u00b0C<\/span><\/span>\n<\/span><\/span>\nbigcore0_thermal-virtual-0<\/span><\/span>\nAdapter: Virtual device<\/span><\/span>\ntemp1:        +79.5\u00b0C<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Die CPU-Temperatur stieg teilweise auf \u00fcber 80\u00b0C, was theoretisch noch im zul\u00e4ssigen Bereich liegt. Dennoch konnte ich ab Temperaturen von etwa 70-80\u00b0C eine Zunahme der fehlerhaften Rechenoperationen beobachten.<\/p>\n\n\n\n
\n
The SoC (RK3588S) is specified to limit its maximum internal temperature to 80\u00b0C before
throttling the clock speeds to maintain reliability within the allowed temperature range. If
the ROCK 5A is intended to be used continuously in high performance applications, it may
be necessary to use external cooling methods (for example, heat sink, fan, etc.) which will
allow the SoC to continue running at maximum clock speed indefinitely below its prede\u2011
fined 80\u00b0C peak temperature limiter.<\/p>\n<\/blockquote>\n\n\n\n
Quelle: https:\/\/docs.rs-online.com\/a9f0\/A700000010117420.pdf<\/a><\/p>\n\n\n\n
Ein erneuter Stresstest mit einem zus\u00e4tzlichen externen L\u00fcfter senkte die SoC-Temperatur auf unter 50\u00b0C, und die Fehler verschwanden vollst\u00e4ndig.<\/p>\n\n\n\n
Fazit<\/h2>\n\n\n\n
Offenbar funktioniert mein ROCK 5 A<\/a> zwar innerhalb der angegebenen Temperaturspezifikation, aber zus\u00e4tzliche K\u00fchlung ist erforderlich, um ein fehlerfreies Arbeiten zu gew\u00e4hrleisten. Der Fehler war besonders interessant, da er auf den ersten Blick nicht auf ein Hardwareproblem hindeutete und die resultierenden Fehlermeldungen sehr unspezifisch waren.<\/p>\n\n\n\n
Den Fehler habe ich beim H\u00e4ndler RS Components beanstandet und problemlos ein Ersatzger\u00e4t f\u00fcr meinen defekten ROCK 5 A<\/a> erhalten. Mit dem neuen ROCK 5 A habe ich keinerlei Probleme mehr.<\/p>\n","protected":false},"excerpt":{"rendered":"
A ROCK 5 A has recently been running as a node in my home lab Kubernetes (k8s) cluster. Gitlab CI jobs with container builds (Buildah, Kaniko) are run on it, for example. After some debugging and extensive troubleshooting, an unusually high error rate in the CI\/CD system turned out to be a hardware error. After several years of experience with Gitlab CI, this was...<\/p>\n
Read More<\/a><\/p>","protected":false},"author":1,"featured_media":446,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43,42,39],"tags":[48,47,49,34],"class_list":["post-345","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-build-automation","category-continuous-deployment","category-continuous-integration","tag-cd","tag-ci","tag-ci-cd","tag-gitlab"],"yoast_head":"\nGitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter<\/title>\n<meta name=\"description\" content=\"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter\" \/>\n<meta property=\"og:description\" content=\"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/\" \/>\n<meta property=\"og:site_name\" content=\"Carl-Christian Sautter\" \/>\n<meta property=\"article:published_time\" content=\"2024-09-24T18:25:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-11-23T14:18:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1080\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"chris\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"chris\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/\"},\"author\":{\"name\":\"chris\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#\\\/schema\\\/person\\\/358bfdbce75a7762072fd564be80419d\"},\"headline\":\"GitLab CI\\\/CD vs. Hitze: Wie ein Hardwarefehler CI\\\/CD Jobs sabotierte\",\"datePublished\":\"2024-09-24T18:25:10+00:00\",\"dateModified\":\"2024-11-23T14:18:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/\"},\"wordCount\":782,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#\\\/schema\\\/person\\\/358bfdbce75a7762072fd564be80419d\"},\"image\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/ci_cd_nodes_burning.jpg\",\"keywords\":[\"cd\",\"ci\",\"CI\\\/CD\",\"gitlab\"],\"articleSection\":[\"Build Automation\",\"Continuous Deployment\",\"Continuous Integration\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/\",\"url\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/\",\"name\":\"GitLab CI\\\/CD vs. Hitze: Wie ein Hardwarefehler CI\\\/CD Jobs sabotierte - Carl-Christian Sautter\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/ci_cd_nodes_burning.jpg\",\"datePublished\":\"2024-09-24T18:25:10+00:00\",\"dateModified\":\"2024-11-23T14:18:39+00:00\",\"description\":\"Wie sich eine hohe Fehlerrate in Gitlab CI\\\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/ci_cd_nodes_burning.jpg\",\"contentUrl\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2024\\\/09\\\/ci_cd_nodes_burning.jpg\",\"width\":1080,\"height\":1080,\"caption\":\"Gitlab CI\\\/CD nodes overheat\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.c-sautter.de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"GitLab CI\\\/CD vs. Hitze: Wie ein Hardwarefehler CI\\\/CD Jobs sabotierte\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#website\",\"url\":\"https:\\\/\\\/www.c-sautter.de\\\/\",\"name\":\"Carl-Christian Sautter\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#\\\/schema\\\/person\\\/358bfdbce75a7762072fd564be80419d\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.c-sautter.de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/#\\\/schema\\\/person\\\/358bfdbce75a7762072fd564be80419d\",\"name\":\"chris\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/P1100639-Bearbeitet-Kopie.jpg\",\"url\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/P1100639-Bearbeitet-Kopie.jpg\",\"contentUrl\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/P1100639-Bearbeitet-Kopie.jpg\",\"width\":1657,\"height\":1657,\"caption\":\"chris\"},\"logo\":{\"@id\":\"https:\\\/\\\/www.c-sautter.de\\\/wp-content\\\/uploads\\\/2023\\\/09\\\/P1100639-Bearbeitet-Kopie.jpg\"},\"sameAs\":[\"https:\\\/\\\/www.c-sautter.de\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/carl-christian-sautter\\\/\"],\"url\":\"https:\\\/\\\/www.c-sautter.de\\\/en\\\/author\\\/chris\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","og_locale":"en_US","og_type":"article","og_title":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","og_description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","og_url":"https:\/\/www.c-sautter.de\/en\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","og_site_name":"Carl-Christian Sautter","article_published_time":"2024-09-24T18:25:10+00:00","article_modified_time":"2024-11-23T14:18:39+00:00","og_image":[{"width":1080,"height":1080,"url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","type":"image\/jpeg"}],"author":"chris","twitter_card":"summary_large_image","twitter_misc":{"Written by":"chris","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#article","isPartOf":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"},"author":{"name":"chris","@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"headline":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte","datePublished":"2024-09-24T18:25:10+00:00","dateModified":"2024-11-23T14:18:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"},"wordCount":782,"commentCount":0,"publisher":{"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"image":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"thumbnailUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","keywords":["cd","ci","CI\/CD","gitlab"],"articleSection":["Build Automation","Continuous Deployment","Continuous Integration"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","url":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/","name":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte - Carl-Christian Sautter","isPartOf":{"@id":"https:\/\/www.c-sautter.de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"image":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage"},"thumbnailUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","datePublished":"2024-09-24T18:25:10+00:00","dateModified":"2024-11-23T14:18:39+00:00","description":"Wie sich eine hohe Fehlerrate in Gitlab CI\/CD nach intensivem Debugging und umfassender Fehlersuche als Hardwarefehler herausstellte.","breadcrumb":{"@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#primaryimage","url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","contentUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2024\/09\/ci_cd_nodes_burning.jpg","width":1080,"height":1080,"caption":"Gitlab CI\/CD nodes overheat"},{"@type":"BreadcrumbList","@id":"https:\/\/www.c-sautter.de\/gitlab-ci-wie-ein-hardwarefehler-ci-cd-jobs-sabotierte\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.c-sautter.de\/"},{"@type":"ListItem","position":2,"name":"GitLab CI\/CD vs. Hitze: Wie ein Hardwarefehler CI\/CD Jobs sabotierte"}]},{"@type":"WebSite","@id":"https:\/\/www.c-sautter.de\/#website","url":"https:\/\/www.c-sautter.de\/","name":"Carl Christian Sautter","description":"","publisher":{"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.c-sautter.de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/www.c-sautter.de\/#\/schema\/person\/358bfdbce75a7762072fd564be80419d","name":"Chris","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg","url":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg","contentUrl":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg","width":1657,"height":1657,"caption":"chris"},"logo":{"@id":"https:\/\/www.c-sautter.de\/wp-content\/uploads\/2023\/09\/P1100639-Bearbeitet-Kopie.jpg"},"sameAs":["https:\/\/www.c-sautter.de","https:\/\/www.linkedin.com\/in\/carl-christian-sautter\/"],"url":"https:\/\/www.c-sautter.de\/en\/author\/chris\/"}]}},"_links":{"self":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/comments?post=345"}],"version-history":[{"count":19,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345\/revisions"}],"predecessor-version":[{"id":448,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/posts\/345\/revisions\/448"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/media\/446"}],"wp:attachment":[{"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/media?parent=345"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/categories?post=345"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.c-sautter.de\/en\/wp-json\/wp\/v2\/tags?post=345"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}