10.5  Acceptance Criteria and Definitions of Done

An acceptance criterion is a measurable condition used to judge whether a requirement is satisfied. It should state environment, stimulus, expected output, tolerance, and evidence. A team’s definition of done should additionally require reviewed code, tests, documentation, configuration, security checks, and integration into the main branch.

Example:

Given a running gateway with 200 queued records and the broker disconnected for 10 minutes, when connectivity is restored, all records shall arrive in sequence within 60 seconds, no database identity shall occur more than once, and queue depth shall return to zero. Evidence: broker log, gateway metric, database query, and test timestamp.

10.6  Traceability

Traceability connects engineering artifacts so a reviewer can move in both directions: from a stakeholder need to its evidence, and from code or a test back to the requirement it supports. Traceability prevents impressive demonstrations from hiding untested requirements and prevents unnecessary features from consuming project time.

Status should mean something precise: proposed, approved, implemented, verified-pass, verified-fail, deferred, or not applicable. “Done” must not mean that code exists without evidence.

10.7  Architecture and Responsibility Allocation

Architecture defines components, their responsibilities, interfaces, constraints, and deployment. A useful diagram distinguishes physical devices, processes or containers, communication links, external services, data stores, users, and trust boundaries.

For every component state:

• inputs, outputs, and owned state;
• deadlines and update rates;
• authority over physical actions;
• behavior during dependency failure;
• configuration and secrets;
• deployment and update method;
• health indicators and logs; and
• responsible team member and reviewer.

Figure 10.1. A deployment diagram should expose physical and trust boundaries rather than showing an unexplained cloud of software.

Avoid the god process: one application that acquires every sensor, controls every actuator, stores data, serves the dashboard, and manages updates. It is easy to demonstrate initially but difficult to isolate, test, secure, and recover. Excessive microservices are also harmful. Choose boundaries that isolate different rates, privileges, failure modes, deployment cycles, or responsibilities.

10.8  Interface Contracts

An interface contract is an agreement between a producer and consumer. For every boundary define:

Consumers must reject unsupported schemas and physically impossible values rather than guessing. Units belong in the contract, not in team memory. A sequence counter must remain unique across restart or be combined with a session identifier. A receiver-reported accuracy field also needs a defined confidence level or vendor uncertainty convention; it is not a measured ground-truth error. Add consumer-driven contract tests so both sides detect breaking changes before field integration.

10.9  Integration Strategy

Big-bang integration combines all completed components late. It produces many simultaneous failures with unclear causes. Prefer continuous vertical integration:

1. define the contract;
2. build a simulator or stub for the missing side;
3. integrate one path end to end;
4. instrument the boundary;
5. automate a small test;
6. replace stubs with real components; and
7. add failure and recovery cases.

Use a known-good test message, recorded sensor data, loopback fixture, or protocol simulator. Preserve one golden end-to-end path while adding features. Integration freezes should stabilize interfaces, not prevent defect fixes.

10.10  Configuration and Build Reproducibility

Source code alone does not reproduce an embedded system. Record:

• operating-system image and architecture;
• package and container versions;
• application commit or release tag;
• firmware, bootloader, and hardware revision;
• schemas and configuration versions;
• required kernel interfaces and permissions;
• calibration coefficients; and
• deployment command and expected health checks.

Configuration belongs under version control when it is not secret. Secrets require separate protected storage and documented provisioning. Pin important dependency versions. A lock file is evidence of inputs, not proof that dependencies are secure or available forever.

10.11  Security Engineering Fundamentals

Cybersecurity protects confidentiality, integrity, and availability while supporting authentic identity, accountability, and recovery. In a cyber-physical system, integrity and availability can have direct physical consequences.

Asset: Something requiring protection, such as actuator authority, location history, keys, firmware, or service availability.

Threat actor: A person, group, service, or compromised device capable of causing harm. Not every threat is malicious; mistakes and faulty automation matter too.

Vulnerability: A weakness that can be exploited or triggered.

Threat: A circumstance or event that could exploit a vulnerability and cause harm.

Risk: The combination of uncertainty, likelihood, exposure, and consequence used to prioritize action.

Attack surface: Reachable interfaces and behaviors through which a system can be influenced.

Trust boundary: A boundary across which identity, data, or authority changes trust assumptions.

Control: A safeguard that prevents, detects, limits, responds to, or supports recovery from a threat.

Security is not achieved by adding TLS at the end. Correctly configured TLS protects confidentiality and integrity in transit and usually authenticates the server; client identity requires an additional authentication mechanism or mutual TLS. It does not make a compromised endpoint trustworthy, validate physical plausibility, authorize every operation, protect stored secrets, or guarantee availability.[38, 76]

10.12  Threat Modeling Step by Step

A lightweight threat-modeling session can follow this sequence:

1. define system scope and unacceptable consequences;
2. inventory assets and safety-relevant authority;
3. draw data flows, external actors, entry points, and trust boundaries;
4. ask how identity, data, commands, availability, and updates can be abused;

5. record preconditions and consequences;
6. select preventive, detective, responsive, and recovery controls;
7. assign an owner and verification test; and
8. revisit the model after architecture changes.

STRIDE is one useful prompt:

STRIDE is a checklist, not a proof of completeness. Physical access, supply-chain compromise, unsafe operator behavior, radio jamming, and environmental failure may require additional analysis.

10.13  Worked Threat Register

Do not record secrets in the threat model. Refer to credential identities or storage locations, never key values.

10.14  Security Controls in Practice

Core controls for student embedded projects include:

• unique device and user identities;
• removal of default credentials;
• least-privilege accounts, files, topics, and API permissions;
• input validation at every trust boundary;
• protected secrets and a rotation/revocation plan;
• authenticated and integrity-checked software updates;
• encrypted communication where exposure justifies it;
• network segmentation and minimal listening services;
• bounded payloads, queues, retries, storage, and execution time;
• actionable logs without passwords, tokens, or unnecessary personal data;
• dependency and vulnerability inventory; and
• tested backup, rollback, and recovery.

10.15  Safety and Security Are Related but Different

Safety addresses unacceptable harm, including harm caused by random faults and foreseeable misuse. Security addresses intentional and accidental compromise of protected properties. An attacker may exploit a security weakness to create a safety hazard; a safe-state design may limit the resulting consequence.

Ask both:

• What if the component fails randomly?
• What if an intelligent adversary deliberately produces the worst valid-looking input?

An emergency stop, motor-current limit, travel limit, watchdog, pressure relief valve, or maximum command duration should not depend solely on a remote dashboard. Match assurance effort to consequence. An ordinary microcontroller is not automatically a safety controller: a credited safety function also needs appropriate fault detection, independence, hardware design, verification, and lifecycle assurance.

10.16  Failure-Mode Analysis

A failure mode describes how a component or function fails. An FMEA-style review records its local effect, system effect, cause, detection, prevention/mitigation, and action owner. It is most valuable before implementation is frozen.

10.16.1  Risk scoring without false precision

Teams often multiply severity, occurrence, and detection rankings. This can support prioritization but does not create objective truth. Scores are ordinal judgments and different combinations may produce the same number with different consequences. Always retain the scenario description and treat intolerable safety or security consequences explicitly, even if estimated likelihood is low.

A practical risk register contains:

• risk identifier and scenario;
• cause, trigger, and consequence;
• existing controls;
• qualitative likelihood and impact with rationale;
• planned treatment and responsible owner;
• due date and status;
• residual risk after treatment; and
• evidence that the treatment works.

10.17  Fault Trees and Common-Cause Failure

An FMEA begins from component failures. A fault tree begins from an unacceptable top event and reasons backward through combinations of causes. For “uncommanded pump operation,” causes might include a forged network command, stuck relay, software state error, or manual wiring bypass.

Redundancy is ineffective when redundant elements share the same power supply, software defect, network, clock, environmental exposure, or incorrect requirement. This is a common-cause failure. Independence must be argued, not assumed.

                      Illustrative tree. AND requires both causes; OR requires any listed cause.

Figure 10.2. Fault-tree reasoning complements component-by-component failure analysis by beginning with an unacceptable system event.

10.18  Verification and Validation

Verification asks whether the implementation satisfies specified requirements: Did we build the system right? Validation asks whether the resulting system satisfies stakeholder needs in its intended context: Did we build the right system?

Passing unit tests verifies selected software behavior. It does not validate that the sensor placement, user workflow, communication coverage, or safety concept solves the real problem.

10.19  Designing Effective Tests

Every test case should contain:

1. stable test identifier and linked requirement;
2. purpose and scope;
3. exact hardware/software/configuration versions;
4. preconditions and setup;
5. controlled input or stimulus;
6. expected result and numerical tolerance;
7. actual result;
8. evidence location;
9. pass/fail decision; and
10. anomaly, corrective action, and retest reference.

Test the following classes deliberately:

• nominal behavior;
• minimum, maximum, and threshold boundaries;
• malformed, unsupported, or physically impossible input;
• missing, stale, delayed, duplicated, reordered, and burst data;
• disconnection and reconnection;
• process, container, and host restart;
• power loss during writing or update;
• clock error and time resynchronization;
• queue and disk exhaustion;
• unauthorized and over-privileged requests;
• long-duration resource and thermal behavior; and
• rollback from a deliberately bad release.

10.20  A Practical Verification Matrix

The statuses above are illustrative examples, not measured results from supplied hardware. A failed test is useful evidence if the team records the failure honestly, identifies the cause, corrects it, and repeats the test. Concealing failures removes the strongest evidence of engineering learning.

10.21  Measurement Quality

Test instruments and references also have uncertainty. Record sampling rate, time synchronization, reference accuracy, calibration status, environmental conditions, and analysis method. Do not report more numerical precision than the measurement supports.

Repeated trials reveal variability. For latency, report median and a high percentile as well as maximum; for packet delivery, report attempted, received, invalid, duplicate, and late counts. Define the denominator. “99% reliable” is meaningless without observation duration, conditions, and failure definition.

10.22  Fault Injection and Recovery Verification

Fault injection deliberately creates controlled failures to test detection and recovery. Examples include blocking a network route, stopping a service, disconnecting a sensor, corrupting a frame, filling a test partition, supplying an expired command, and restarting during a write.

Perform fault injection only on a safe test setup. Define the abort condition and protect equipment. Never create electrical faults, unsafe motion, or uncontrolled load merely for demonstration.

Measure:

• fault-detection time;
• transition to degraded or safe state;
• data lost, duplicated, or reordered;
• restart and service-restoration time;
• operator notification and diagnostic value; and
• whether manual intervention is required.

10.23  Security Verification

Security tests should include expected rejection, not only successful access:

• invalid, expired, and revoked credentials;
• a valid identity attempting an unauthorized topic or operation;
• oversized and malformed payloads;
• replayed command identifiers or sequence numbers;
• unexpected listening ports;
• secret scanning of repository and release artifacts;
• file and device permissions;
• update with invalid signature or incompatible version;
• log inspection for secret leakage; and
• resource exhaustion within a controlled environment.

An automated vulnerability scanner cannot determine whether a valve is left open after network loss. Security verification must include application and physical consequences.

10.24  Deployment and Rollback

A deployable release is a traceable set of code, configuration, dependencies, schemas, firmware, and instructions. Deployment should be repeatable and preferably automated. Before release:

1. tag or identify the approved revision;
2. build from a clean environment;
3. run required tests;
4. archive checksums and dependency inventory;
5. back up required data and configuration;
6. state compatibility and migration steps;
7. define health checks and observation period; and
8. prepare and test rollback.

Rollback must preserve compatible configuration and data. Reinstalling an earlier binary may fail after an irreversible database or firmware migration. Where possible, use backward-compatible staged migrations and retain a known-good system image.

10.25  Observability and Operational Acceptance

The project is not finished when it works while its developers watch a terminal. An operator needs to know:

• whether every critical component is alive and ready;
• when each measurement was physically observed;
• queue depth and data age;
• current software/configuration versions;
• resource and storage state;
• recent restarts and reset reasons;
• degraded modes and inhibited actions; and
• what corrective action an alert requires.

Alert on actionable conditions. Too many non-actionable alarms train users to ignore the system.

10.26  Team Integration and Individual Accountability

Teamwork does not remove individual assessment. Use a responsibility matrix and evidence map. Each interface needs a producer owner, consumer owner, and reviewer. Each student should contribute to design, implementation, verification, and explanation rather than owning only slides or documentation.

Useful attribution includes:

• meaningful commits and reviewed pull requests;
• dated design decisions;
• test cases executed and analyzed by the student;
• authored interface or threat entries;
• recorded debugging and correction;
• individual explanation during demonstration; and
• reflection connecting evidence to course learning objectives.

Commit count and lines of code are not competence measures. A student may make a small but critical architectural or verification contribution. The portfolio must explain the intellectual and practical work.

10.27  Evidence Quality for the Portfolio

Strong evidence answers four questions:

1. Context: What requirement, component, version, and test setup is shown?
2. Action: What did the student design, implement, measure, or correct?
3. Result: What observable output was obtained, including failures?
4. Meaning: What does the evidence prove, and what does it not prove?

10.28  Formal Integration Review

The Module 10 review should contain:

1. one-sentence problem and intended user;
2. current measurable requirements and priorities;
3. deployment architecture with trust and safety boundaries;
4. interface contracts and versioning;
5. live thin vertical slice;
6. current traceability and verification matrices;
7. principal security threats and failure modes;
8. risk register with owners and planned treatment;
9. deployment, health, and rollback approach;
10. prioritized backlog to final demonstration; and
11. individual portfolio evidence map.

Reviewers should challenge undefined adjectives, hidden units, unbounded retries, missing clocks, shared credentials, absent safe states, unverified recovery, vague ownership, and claims without evidence.

10.29  Self-Assessment and Job-Interview Questions

These questions assess course learning while preparing students for embedded Linux, IoT, robotics, systems, security, test, and integration interviews. A strong answer identifies assumptions and gives a concrete example.

1. What is a thin vertical slice, and why is it valuable?
2. Rewrite “the system shall be reliable” as a testable requirement.
3. Distinguish a stakeholder need, requirement, architecture decision, and acceptance criterion.
4. What is bidirectional traceability?
5. What information belongs in an interface contract?
6. Why are units and time semantics integration concerns?
7. Compare big-bang and continuous integration.
8. Define asset, vulnerability, threat, control, and risk.
9. What is a trust boundary?
10. Why does TLS not solve all security problems?
11. Apply STRIDE to an MQTT-controlled irrigation valve.
12. Why should each physical command carry an identifier and expiry?
13. What is least privilege in an embedded Linux project?
14. Distinguish safety and cybersecurity.
15. What is a failure mode, and how does FMEA help?
16. What is a common-cause failure?
17. Distinguish verification and validation.
18. Compare unit, integration, system, and acceptance tests.

19. Why are negative tests important?
20. What makes a test repeatable?
21. How would you test a ten-minute broker outage?
22. Why is average latency alone insufficient?
23. What must a rollback plan preserve?
24. Why are screenshots weak evidence by themselves?
25. How can individual contribution be demonstrated fairly in a team project?
26. Is a failed test acceptable in a portfolio?
27. What should the Module 10 review produce?

10.30  Answers and Discussion

1. A thin vertical slice is the smallest useful path crossing real layers from physical input to useful output, possibly including command feedback. It reveals boundary, deployment, timing, identity, and ownership problems earlier than completing isolated subsystems.
2. One example is: “During a 24-hour test, the gateway shall deliver at least 99% of valid one-minute measurements to storage, excluding documented maintenance, and shall recover automatically from each broker interruption shorter than five minutes.” The target still requires stakeholder justification.
3. A need describes stakeholder value; a requirement states mandatory observable behavior or quality; an architecture decision chooses how responsibilities are allocated; an acceptance criterion defines evidence for judging satisfaction.
4. Forward traceability connects a need and requirement to implementation and test. Backward traceability connects code, tests, and features to the requirement or decision that justifies them. Both directions reveal missing evidence and unnecessary work.
5. Include producer, consumer, transport, endpoint, schema/version, types, units, identity, timing, size/rate, authentication, authorization, timeout, retry, acknowledgement, duplicate/order behavior, errors, compatibility, and owner.
6. A temperature value can differ by unit; a timestamp may mean acquisition or receipt; clocks may be unsynchronized. Without explicit semantics, syntactically successful communication can create physically wrong decisions.
7. Big-bang integration combines many components late and makes fault isolation difficult. Continuous integration connects small vertical paths repeatedly, uses stubs where necessary, and adds automated boundary tests.
8. An asset requires protection. A vulnerability is a weakness. A threat is a circumstance that can exploit a weakness. A control reduces, detects, responds to, or supports recovery. Risk expresses uncertainty and consequence for prioritization.
9. A trust boundary is a location where identity, data, authority, or assumptions change, such as from a field radio network to a gateway or from an authenticated API to actuator control. Input must be revalidated there.
10. TLS can protect data in transit and authenticate endpoints when configured correctly. It does not constrain authorized misuse, repair compromised endpoints, validate physical values, prevent every denial of service, or protect secrets at rest.
11. Spoofing: fake valve controller identity; tampering: altered duration; repudiation: disputed command; disclosure: exposed farm activity; denial of service: command flood; elevation: dashboard account gains broker administration. Controls and tests should be assigned for the credible cases.
12. The identifier supports acknowledgement and deduplication. Enforced expiry rejects commands received after their validity window; duplicates replayed within that window still require deduplication or replay protection. An absolute expiry also requires a trustworthy clock or an explicit policy for time uncertainty.

13. Run services as dedicated non-root users, grant only required device/file/topic/API access, remove unused capabilities and ports, and separate monitoring authority from actuator authority.
14. Safety protects against unacceptable harm from faults and foreseeable behavior; cybersecurity protects confidentiality, integrity, availability, identity, and authority against compromise. They interact when a cyberattack can cause physical harm.
15. A failure mode states how a function fails. FMEA systematically records causes, effects, detection, mitigation, verification, and ownership so important faults are addressed before final testing.
16. A common-cause failure defeats apparently redundant channels through a shared dependency or defect, such as one power supply, clock, library, cable route, or incorrect requirement.
17. Verification checks conformance to specified requirements; validation checks whether the system meets real stakeholder needs in its intended environment.
18. A unit test isolates a small software element; an integration test crosses component boundaries; a system test exercises the assembled system; an acceptance test judges stakeholder-facing criteria.
19. Successful tests show intended access and nominal behavior. Negative tests show that invalid data, unauthorized clients, expired commands, and unsupported versions are safely rejected.
20. A repeatable test identifies versions, setup, preconditions, stimulus, expected result, tolerances, measurement method, actual result, and evidence. Another person should be able to reproduce it.
21. Establish known sequence counts, disconnect the broker, continue acquisition, observe bounded queue growth, reconnect, and verify replay time, missing/duplicate identities, ordering policy, queue recovery, logs, and alerts.
22. Average latency hides tail delays and missed deadlines. Report distributions, high percentiles, maximum, test load, sample count, clock method, and end-to-end boundaries.
23. Rollback must restore compatible application, firmware, schemas, configuration, secrets references, and data format. It must account for migrations and retain a known-good artifact.
24. A screenshot rarely establishes the input, time, version, ownership, expected value, or correctness. Annotate it and connect it to a requirement, controlled test, result, and limitation.
25. Use attributable commits and reviews, authored designs/contracts, executed tests, debugging records, oral explanation, and reflection. Do not reduce contribution to line or commit count.
26. Yes. An honestly documented failure, diagnosis, correction, and successful retest can demonstrate stronger competence than a context-free success. An unexplained or concealed failure cannot.
27. It should produce agreed requirements and interfaces, an assessed architecture, prioritized threats/failures/risks, a verification and deployment plan, assigned review actions, a realistic backlog, and an individual evidence map.

10.31  Authoritative References and Verification Links

The following primary sources provide further guidance. Standards may require institutional access:

• ISO/IEC/IEEE 29148 systems and software requirements engineering
• ISO/IEC/IEEE 24748-6:2023 guidance for system and software integration
• NIST Cybersecurity Framework 2.0
• NIST Secure Software Development Framework SP 800-218
• NIST Cybersecurity for Internet of Things program
• OWASP Internet of Things project

• OWASP Application Security Verification Standard
• CISA Secure by Design guidance
• FIRST Common Vulnerability Scoring System