11.7  Logs That Support Diagnosis

Useful logs are structured events rather than prose dumps. Include:

• synchronized timestamp and component identity;
• severity and stable event name;
• release/configuration version;
• device, operation, sequence, or correlation identifier;
• relevant state and bounded diagnostic context; and
• outcome and recoverability.

Never log passwords, tokens, private keys, full authorization headers, or unnecessary personal data. Debug logging can change timing and fill storage; enable it deliberately and remove it after investigation.

11.8  systemd and Journal Diagnosis

Inspect service configuration with systemctl cat and the combined effective properties with systemctl show. After editing a unit, use systemctl daemon-reload. Restarting without reload may continue using the earlier unit definition. A single -p warning includes warning and every more severe priority, including emergency. Previous-boot queries require retained journal data.[13]

11.9  Tracing System Calls

strace observes system calls and signals. It is useful when a program fails to open a file, connect a socket, access a device, or waits unexpectedly.

Tracing changes timing, can produce sensitive output, and may be prohibited on production systems. Filter the call classes and capture for a bounded period. ENOENT means a path was not found; EACCES indicates permission failure; repeated poll or futex calls may reveal waiting but require context.

11.10  Network Debugging by Layers

Use a consistent sequence:

1. verify physical link or radio association;
2. inspect address and link state;
3. inspect routes and selected path;
4. test name resolution separately from reachability;
5. confirm that the expected service is listening;
6. test transport connection;
7. test application protocol and authentication; and
8. inspect return traffic, firewall, and server logs.

The documentation address 192.0.2.10, example host, service names, and PID tokens must be replaced with values for the test system. resolvectl is relevant only where systemd-resolved is in use. A successful ping does not prove that an application port, TLS handshake, credentials, topic permission, or payload is correct. A failed ping does not prove the host is unavailable because ICMP may be filtered.

11.11  Hardware and Serial Debugging

For UART, I²C, SPI, CAN, GPIO, and USB, verify electrical compatibility before software. Typical causes include wrong voltage, missing common reference, swapped lines, absent pull-ups, incorrect termination, floating pins, inadequate power, long cables, EMI, connector intermittency, and incorrect bus parameters.

Bus scanners can disturb certain devices. Never probe an unknown live industrial or safety-relevant bus without understanding the consequences. Use an oscilloscope or logic analyzer to confirm voltage, timing, framing, and noise when software evidence is insufficient.

11.12  Intermittent and Timing Faults

Intermittent bugs often disappear under a debugger. Add lightweight sequence numbers, state transitions, monotonic timestamps, queue depth, and deadline metrics. Compare clocks carefully: wall-clock time can jump during synchronization, while a monotonic clock is better for durations.

Race conditions require stress, repeated execution, and tools appropriate to the language. Do not “fix” a race by adding arbitrary sleep. A sleep may alter probability without establishing synchronization.

11.13  Minimal Reproduction and Binary Search

Reduce the system until the failure disappears, then restore one element. Examples:

• replace a sensor with recorded input;
• replace the cloud with a local known-good broker;
• bypass DNS by testing the resolved address separately;
• run the process outside a container, then restore constraints;
• use git bisect to test selected revisions between a known-good and known-bad commit; or
• compare a known-good hardware node with the failing node while changing one item at a time.

A minimal reproduction should retain the failing mechanism while removing unrelated complexity. Document every removed assumption.

11.14  Reproducible Releases

A release is an identified, tested set of artifacts and metadata intended for deployment. A release is not an unnamed directory copied from a laptop.

Record:

• semantic or project version and source commit;
• build time and controlled build environment;
• target architecture and operating-system baseline;
• dependency lock files or package manifest;
• firmware, bootloader, and hardware compatibility;
• configuration schema and migration requirements;
• database or stored-data schema;
• container image digest or artifact checksum;
• known limitations; and
• verification results and approval.

A checksum detects accidental or malicious modification only if the expected checksum is obtained through a trustworthy channel. A digital signature additionally authenticates the signer when keys are managed correctly.

11.15  Reproducibility Has Levels

Repeatable: The same team can obtain the result using the same setup.
Reproducible: Another qualified person can rebuild or repeat it from documented inputs.
Hermetic build: The build depends only on declared inputs and controlled tooling rather than
    hidden host state or network downloads.
Deterministic build: Identical declared inputs produce byte-identical outputs, where the
    toolchain supports it.

Containers improve environment control but do not automatically make builds deterministic. Unpinned base images, package mirrors, timestamps, architecture differences, and external downloads remain sources of variation.

11.16  Configuration and Secrets

Validate configuration before starting useful work. Fail with a precise error rather than silently inventing a dangerous default.

Do not include secrets in container images, Git, command-line arguments, or general environment dumps. Separate public configuration from protected credentials. Document provisioning, rotation, revocation, and recovery without revealing the secret value.

11.17  Release and Deployment Strategies

For a student system, a versioned directory plus a stable symbolic link can provide simple atomic switching, provided service state and configuration remain compatible.

The temporary link must not already exist, the deployment directory must be writable only by the authorized deployer, and concurrent deployments must be excluded. On this GNU/Linux layout, the same-filesystem rename replaces the link atomically; ln -sfn alone does not provide the same guarantee. This changes a pathname, not the running process, database, or configuration as one transaction. If restart or readiness fails, use the recorded prior target and the rollback procedure. The exact procedure must also handle power-loss durability, ownership, partial failure, and authorization.

11.18  Deployment Checklist

11.19  Smoke Checks and Health Semantics

A smoke test quickly checks the most important release paths. It does not replace full verification. Distinguish:

• startup: required initialization has completed;
• liveness: the component is making enough progress that restart is not justified;
• readiness: it can currently accept useful work;
• health: richer diagnostic state for operators; and
• functional result: the actual end-to-end requirement is satisfied.

A web process returning HTTP 200 from /health can still publish incorrect units. Health checks should be cheap, bounded, and unable to amplify dependency outages.

11.20  Rollback and Data Migration

Define rollback triggers before deployment: repeated crash, failed readiness, unacceptable latency, lost data, incompatible interface, resource growth, or safety-related anomaly. Rollback should be a rehearsed operation rather than improvisation.

Data migration complicates rollback. If version 2 irreversibly transforms the database, version 1 may no longer read it. Prefer additive, backward-compatible changes:

1. deploy code that understands old and new forms;
2. add new schema fields without removing old ones;
3. migrate data while both versions remain compatible;
4. stop writing the old form after observation; and
5. remove obsolete fields only in a later release.

Backups are not proven until restoration is tested. Record recovery-point objective (acceptable data loss) and recovery-time objective (acceptable restoration time) when the project needs them.

11.21  Resource Management

Embedded resources are finite. Observe both current utilization and pressure over time.

Pressure Stall Information reports time during which tasks stall because CPU, memory, or I/O resources are contended. It complements utilization: a CPU can appear busy without harmful delay, while memory or I/O pressure may block critical work.

11.22  Memory Exhaustion and the OOM Killer

When memory cannot be reclaimed, Linux may invoke the out-of-memory mechanism and terminate a selected process. Diagnose with kernel logs rather than assuming an application crash:

Restarting an unbounded process can create a cycle. Bound memory at the design level, configure meaningful service/container limits, expose queue size, and test outage duration beyond expected operation.

11.23  Storage Capacity and Endurance

Storage can be full by bytes or by inodes. Deleted files may continue consuming space while a process holds them open. Investigate with df -h, df -i, du, and lsof +L1. Rotate and retain logs according to an explicit policy.

Flash media also has finite write endurance and is vulnerable to abrupt power interruption. Measure write amplification, batch updates when acceptable, use atomic replacement and suitable filesystems, separate transient from durable data, and choose media appropriate to expected writes and environment.

11.24  Performance Engineering

Begin with a question: “Can the gateway validate and store 50 measurements per second while keeping 95th-percentile end-to-end latency below 500 ms?” Define workload, boundaries, clock, warm-up, duration, hardware, software, and acceptance threshold.

For latencies t₁, …, t_n with n ≥ 2, the sample mean and sample standard deviation are

The p₉₅ latency is a value at or below which approximately 95% of observations fall under the chosen sample definition. State the percentile method and sample count when precision matters. Tail latency often determines missed deadlines and operator experience.

11.24.1   Coordinated omission

A load generator that waits for each slow response before issuing the next request may fail to measure delays that would have occurred under the intended arrival rate. This is coordinated omission. Generate workload according to the real arrival model and record rejected or unserved requests.

11.25  Profiling Before Optimization

Optimization should follow measurement:

1. define the metric and target;
2. establish a repeatable baseline;
3. profile to locate the bottleneck;
4. change one justified mechanism;
5. repeat under identical conditions;
6. evaluate side effects; and
7. retain the simpler design if improvement is not meaningful.

Useful tools include language profilers, perf, timing instrumentation, flame graphs, database query plans, network capture, and power measurement. Optimize the end-to-end path, not merely a convenient function.

Profiling requires suitable permission and can add overhead. Never expose unrestricted profiling access on an operational system.

11.26  Common Optimization Trade-offs

Every portfolio optimization should report its cost, not only its improvement.

11.27  Resilience and Fault-Injection Testing

Test failures systematically:

• kill, terminate, pause, or deadlock a process safely;

• restart services in unexpected order;
• disconnect and restore networking;
• reject authentication or rotate credentials;
• disconnect a sensor or send malformed, stale, duplicate, and reordered data;
• fill a test queue and approach a controlled storage limit;
• supply missing and invalid configuration;
• interrupt an update in a safe test environment;
• reboot and power-cycle the target; and
• run a long soak test with representative workload.

For each fault, verify detection time, state transition, safe physical behavior, data loss/duplication, diagnostic clarity, automatic recovery, operator action, and recovery time.

11.28  Power-Cycle Testing

A software restart does not reproduce abrupt power loss. Power-cycle testing reveals boot ordering, storage durability, peripheral initialization, clock behavior, retained commands, and safe output state. Use low-risk equipment and a controlled switching method. After power restoration verify:

• outputs remain safe during boot;
• file systems and databases are consistent;
• services wait for actual dependencies rather than arbitrary sleep;
• time uncertainty is represented until synchronization;
• queues and sequence identifiers recover correctly; and
• the device returns to a known operational or diagnosable state.

11.29  Soak and Endurance Testing

A soak test operates the system under representative load for an extended period. Track memory, file descriptors, queue depth, storage, CPU, latency percentiles, temperature, reconnects, errors, and output correctness. Plot values over time; a slow leak may be invisible in a short benchmark.

Include environmental and workload cycles. A 24-hour test at constant room conditions does not establish outdoor seasonal reliability, but it can reveal resource leaks and repeated scheduled failures.

11.30  Incident Narrative and Learning

A useful debugging record contains:

1. impact and observable symptom;
2. detection source and timeline;
3. affected versions and scope;
4. evidence preserved;
5. hypotheses and experiments;
6. root cause and contributing factors;
7. immediate containment;
8. permanent corrective action;
9. regression test and monitoring added; and
10. remaining uncertainty and follow-up.

Blameless does not mean accountability-free. It means analyzing system conditions and decisions so recurrence becomes less likely, while clearly assigning corrective actions.

11.31  Guided Laboratory

11.32  Self-Assessment and Job-Interview Questions

These questions assess the course material and prepare students for embedded Linux, robotics, IoT, DevOps, test, and integration interviews.

1. What distinguishes evidence-led debugging from trial and error?
2. Distinguish a symptom, proximate cause, root cause, and contributing factor.
3. Why should evidence be preserved before restarting a failed system?
4. Why is changing several variables at once poor debugging practice?
5. A service is active but measurements are stale. What should you inspect?
6. What do systemctl status and journalctl -u contribute?

7. When is strace useful, and what are its limitations?
8. Give a layered plan for diagnosing a failed HTTPS or MQTT connection.
9. Why can arbitrary sleep appear to fix a race condition?
10. What is a minimal reproduction?
11. What information uniquely identifies a reproducible release?
12. Compare a checksum and a digital signature.
13. Why does a container not guarantee a reproducible build?
14. Distinguish liveness, readiness, and functional correctness.
15. What is a smoke test?
16. Compare in-place, A/B, blue/green, canary, and rolling deployment.
17. What makes rollback difficult after a data migration?
18. How would you diagnose Linux OOM termination?
19. Why must both bytes and inodes be checked for storage exhaustion?
20. What does Pressure Stall Information add to utilization metrics?
21. Why is mean latency insufficient?
22. What is coordinated omission?
23. Describe a defensible optimization experiment.
24. Why can batching improve throughput but worsen reliability or latency?
25. What should a network-outage resilience test measure?
26. Why is a controlled power cycle different from restarting a service?
27. What does a soak test reveal?
28. What makes a debugging narrative strong portfolio evidence?

11.33  Answers and Discussion

1. Evidence-led debugging states the symptom precisely, forms competing hypotheses, predicts observations, and runs discriminating tests. Trial and error changes things without preserving causal knowledge.
2. The symptom is observed behavior; the proximate cause directly creates it; the root cause is an underlying correctable condition whose removal prevents recurrence in scope; a contributing factor increases probability or impact.
3. Restarting destroys volatile state, timing, process information, queues, and sometimes logs. Preserve the evidence required to reconstruct what happened, unless immediate safety requires shutdown.
4. If the result changes, the responsible variable is unknown and interactions are hidden. Change one relevant factor or use a designed experiment when interactions are the question.
5. Check source observation timestamps and sequence progress, acquisition health, queues, threads, resource pressure, device input, dependency status, logs, and readiness semantics. Process existence proves little.
6. systemctl status summarizes current unit state and recent messages; journalctl -u provides filterable service history across a defined boot/time interval. Effective unit properties and kernel logs may also be useful.
7. strace reveals system calls, errors, signals, file access, sockets, and waits. It adds overhead, changes timing, may expose sensitive data, needs permission, and does not directly explain application-level intent.
8. Verify link, address, route, DNS, listening endpoint, transport connection, TLS identity/time, protocol response, authentication, authorization, payload/schema, return traffic, and server logs. Each step answers a narrower question.
9. Sleep changes scheduling probability and may hide a missing synchronization mechanism. It is workload-dependent and does not establish ordering, mutual exclusion, or readiness.

10. A minimal reproduction retains the failing mechanism with the fewest relevant components and explicit steps. It speeds experiments and makes hidden assumptions visible.
11. Record source commit/tag, dirty state, artifact checksum/digest, build environment, target architecture/OS, dependencies, firmware/hardware compatibility, configuration and data schemas, migrations, and verification result.
12. A checksum detects different bytes when the expected value is trusted. A digital signature links an artifact to a private signing key and can authenticate origin when key trust and protection are valid.
13. Base tags can move; packages and downloads change; host kernel/architecture remain external; timestamps and build tools vary; secrets/configuration may be injected. All inputs must be declared and controlled.
14. Liveness asks whether restart is justified, readiness whether useful work can currently be accepted, and functional correctness whether the required output is actually correct under stated conditions.
15. A smoke test is a fast post-build or post-deployment check of critical paths. It detects gross release failure but does not replace complete verification.
16. In-place modifies the active installation; A/B switches between system slots; blue/green keeps complete old/new service environments; canary exposes a small cohort first; rolling replaces instances gradually. Compatibility and rollback needs differ.
17. Old code may not understand a new schema, and the migration may discard or transform data irreversibly. Use staged backward-compatible migrations and test restoration and rollback.
18. Inspect kernel/journal messages for OOM and killed-process records, service results, memory/cgroup limits, RSS and trends, swap/pressure, queues, and allocation profiles. Correct leaks or unbounded behavior rather than only raising limits.
19. A filesystem can have free bytes but no available inodes, preventing new files. Conversely, large files can exhaust capacity while inodes remain. Open deleted files can consume space invisibly to ordinary du.
20. PSI measures time tasks are stalled by CPU, memory, or I/O contention. It helps distinguish harmless utilization from resource pressure that delays useful work.
21. The mean hides variability and rare delays. Report sample count, distribution, median, high percentiles, maximum, missed deadlines, workload, and end-to-end measurement boundaries.
22. Coordinated omission occurs when the measurement workload slows with the system and fails to generate requests that should have arrived during long responses, understating user-visible delay.
23. Define target and conditions, establish a repeatable baseline, profile the bottleneck, change one mechanism, rerun the same workload, quantify uncertainty and tail behavior, and report costs and regressions.
24. Batching amortizes overhead but waits for a batch to fill and places more data at risk in one failed operation. It may increase memory and recovery complexity.
25. Measure detection, queue growth and bounds, local behavior, data accepted/lost/discarded, reconnect backoff, replay time, duplicates/order, service readiness, alerting, and complete recovery.
26. Service restart retains kernel, device, filesystem, clock, and much system state. Power cycling tests boot ordering, safe outputs, durable storage, peripheral initialization, retained data, and time synchronization.
27. A soak test reveals slow memory/file-descriptor leaks, queue accumulation, storage growth, thermal throttling, periodic faults, reconnect patterns, and long-tail performance under representative load.
28. It links a precise symptom and version to preserved evidence, competing hypotheses, discriminating tests, root cause, correction, regression prevention, remaining uncertainty, and the student’s own contribution.

11.34  Authoritative References and Verification Links

Tool behavior and platform capabilities evolve. Verify commands and configuration against primary documentation:

• systemctl manual
• journalctl manual
• systemd service-unit documentation
• strace manual page
• tcpdump manual
• Linux kernel control-group v2 documentation
• Linux Pressure Stall Information documentation
• Linux perf documentation
• Docker container resource constraints
• Dockerfile health-check reference
• Git tag documentation
• Git bisect documentation