CourseFall 2026

Friday · 12:00–16:00 · Odense U174

Distributed Networking, Web Services, Monitoring, and Remote Management

Lecture 03: This module develops a practical engineering approach to distributed embedded Linux systems. It covers distributed-system differences, latency, availability and consistency, identity and discovery, protocol layers, HTTP application design, measurement data contracts, validated web services, timeouts and retries, idempotency and deduplication, offline operation, observability, health endpoints, dashboards, alerts, and secure remote management.

Learning objectives

✓By the end of this module, students should be able to:

✓• explain partial failure, uncertainty, latency, duplication, ordering, and network partitioning;

✓• separate service identity from address, location, and service endpoint;

✓• design a small resource-oriented HTTP API with meaningful methods and status codes;

✓• specify a versioned measurement contract with units, timestamps, quality, and validation;

✓• choose timeout, retries, backoff, idempotency, deduplication, and offline buffering deliberately;

✓• distinguish logs, metrics, traces, health checks, dashboards, and alerts;

✓• identify trust boundaries and protect remote administration and software updates; and

✓• test distributed embedded services under delay, loss, duplication, dependency failure, and recovery.

Block 112:00–12:50content

Block 1: 12:00–12:50 · Distributed Systems Architecture and Communication

CHAPTER 3

Distributed Networking, Web Services,
Monitoring, and Remote Management

Learning objectives

By the end of this module, students should be able to:

explain partial failure, uncertainty, latency, duplication, reordering, and network partitioning;
separate persistent device identity from address, location, and service endpoint;
design a small resource-oriented HTTP API with meaningful methods and status codes;
specify a versioned measurement contract with units, timestamps, quality, and validation rules;
choose timeouts, retries, backoff, idempotency, deduplication, and offline buffering deliberately;
distinguish logs, metrics, traces, health checks, dashboards, and alerts;
identify trust boundaries and protect remote administration and software updates; and
test a distributed embedded service under delay, loss, duplication, dependency failure, and recovery.

Why distributed systems require different reasoning

When a local function returns, the caller normally knows whether it completed. Across a network, the request, response, peer, route, name service, or authentication system may fail independently. If a client times out, it does not know whether the server never received the request, completed it but lost the response, or is still working. Robust distributed systems make this uncertainty explicit instead of hiding it behind optimistic code.

3.1 The Distributed-Systems Difference

A distributed system contains independently executing components that communicate through channels with finite capacity, variable delay, and possible interruption. Each component observes only part of the total state. There is no instantaneous, perfectly reliable global view.

24-Embedded Linux – Fall 2026

Figure 3.1. An embedded node, Linux gateway, application service, storage, and operator interface form a distributed system. The local queue supports buffering at the gateway; every link and service remains a possible partial-failure boundary.

25-Embedded Linux – Fall 2026

Remote communication can experience:

omission: a request or response is lost;
delay: delivery takes longer than expected;
duplication: the same logical operation arrives more than once;
reordering: messages arrive in a different order;
corruption: bits or application data are invalid, despite lower-layer checks;
partition: groups of nodes remain operational but cannot communicate;
crash and restart: a node loses volatile state and later returns; and
Byzantine or compromised behavior: a component returns misleading or malicious information.

The partial-failure principle

One component can fail while others continue. Every external dependency needs a bounded wait, a defined degraded mode, visible state, and a recovery or reconciliation path. “Wait forever” and “retry forever” transfer failure into resource exhaustion; they do not provide recovery.

3.2 Latency, Availability, and Consistency

Latency is not a constant; it is a distribution. A system that is usually fast may have a long tail caused by retransmission, radio interference, scheduling, storage, DNS, or overloaded dependencies. Report percentiles and worst observed values rather than only an average.

Availability is the fraction of time a service satisfies its defined objective. A component being powered does not make the application available. Define what operation must succeed, from where, within what time, and with what freshness.

During a network partition, a design may accept local work and reconcile later, or reject work that cannot be coordinated safely. Neither choice is universally correct. Environmental monitoring can often buffer measurements offline. Coordinated actuation may require stronger rules because conflicting commands can cause physical harm.

Workload	Reasonable degraded mode	Important recovery question
Temperature monitoring	Store timestamped samples locally	How are duplicates and gaps identified after upload?
Dashboard display	Show last value with age and offline state	When must stale data stop being presented as current?
Configuration update	Reject or queue until authority is reachable	Which version wins after reconnection?
Irrigation command	Continue only under validated local safety policy	How are late or duplicated commands prevented?
Robot supervision	Enter a known-safe local state on loss of authority	How is control ownership re-established?

26-Embedded Linux – Fall 2026

3.3 Identity, Addressing, Naming, and Discovery

These concepts must not be collapsed:

identity answers which logical device or service this is;
address identifies a reachable network interface at a moment in time;
name is a human- or system-meaningful reference resolved to other information;
endpoint combines protocol, address or name, and port; and
discovery finds currently available services and their endpoints.

A persistent identifier should remain stable when DHCP changes an address, a device moves to another network, or a service is replicated. Do not use a mutable IP address as the primary database identity. Conversely, a printed device identifier does not prove that the communicating device is authentic; cryptographic identity is a separate security property.

Discovery choices include configured endpoints, DNS, multicast DNS, service registries, and broker-based rendezvous. Static configuration is simple but difficult to change across a fleet. Multicast DNS is convenient on a local link but should not be assumed to cross routers. A registry supports dynamic systems but introduces another dependency and trust boundary.

3.4 Protocol Layers and End-to-End Diagnosis

An application message crosses several conceptual layers. Ethernet or Wi-Fi transfers frames on a local network; IP routes packets; TCP or UDP provides transport behavior; TLS can add peer authentication, confidentiality, and integrity; HTTP defines request and response semantics; the application defines resources and data meaning.

Layer	Example evidence	Question answered
Link	ip -brief link	Is the interface present and administratively operational?
Network	ip address, ip route	Which source address and next hop will be used?
Naming	getent hosts api.example	What result does the configured resolver return?
Transport	ss, nc -vz	Is a socket listening or a TCP connection possible?
TLS	openssl s_client or curl -v	Which certificate and negotiation result are observed?
HTTP	status, headers, body	Did the request reach an HTTP application and how did it respond?
Application	logs, metrics, stored record	Did the intended operation occur correctly and durably?

Success at one layer does not prove success above it. A successful ping does not prove that HTTPS is listening; a TCP connection does not prove authorization; HTTP 200 does not prove that returned sensor data are fresh or correct.

27-Embedded Linux – Fall 2026

Block 213:00–13:50content

Block 2: 13:00–13:50 · HTTP, Web Services, and Measurement Data Contracts

HTTP is a request-response protocol. A request contains a method, target, headers, and optional body. A response contains a status code, headers, and optional body. HTTP semantics matter more than the framework used to implement them [37].

Method	Usual intent	Design note
GET	Retrieve a representation	Safe and idempotent; must not perform hidden actuation
POST	Submit data or create subordinate resource	Repetition may create duplicates unless protected
PUT	Create or replace a known resource	Intended to be idempotent
PATCH	Partially modify a resource	Semantics and conflict handling must be specified
DELETE	Request resource removal	Intended effect should be idempotent

Status-code families communicate result categories. Use 2xx for successful handling, 4xx when the request is unacceptable, and 5xx when the server cannot fulfill an otherwise meaningful request. Useful distinctions include 201 Created, 204 No Content, 400 Bad Request, 401 Unauthorized (authentication required or failed), 403 Forbidden (the server refuses the request; authentication may or may not have succeeded), 404 Not Found, 409 Conflict, 413 Content Too Large, 422 Unprocessable Content, 429 Too Many Requests, 503 Service Unavailable, and 504 Gateway Timeout.

A resource-oriented API exposes stable domain concepts:

Listing 3.1. Example resource paths and operations

1GET  /api/v1/devices/device-0172GET  /api/v1/devices/device-017/measurements/latest3POST /api/v1/devices/device-017/measurements4GET  /api/v1/operations/op-8f315PUT  /api/v1/devices/device-017/configuration

Avoid encoding internal function names into URLs. Version interfaces deliberately and define compatibility policy. A version number does not replace change discipline.

Figure 3.2. A user-creation example illustrates HTTP request and response metadata, a JSON body, and an application-defined correlation identifier. A successful creation returns status 201 and a resource location. The token is a dummy value; the measurement service below defines a separate application contract.

3.6 Designing a Measurement Data Contract

JSON is readable and widely supported, but syntax alone does not define meaning. A contract must specify required fields, types, allowed ranges, units, timestamp representation, identifier rules, missing-value policy, quality indicators, schema version, and error behavior.

Listing 3.2. A versioned measurement representation

1{2  "schema_version": 1,3  "device_id": "device-017",4  "measurement_id": "01J7M6FJ9R8K2Q3S4T5V6W7X8Y",5  "observed_at": "2026-09-18T12:04:31.420Z",6  "received_at": "2026-09-18T12:04:31.106Z",7  "values": {8    "air_temperature": {"value": 22.7, "unit": "degC"},9    "relative_humidity": {"value": 63.1, "unit": "%"}10  },11  "quality": "valid",12  "sequence": 184213}

Observation time and receipt time answer different questions. The server should not silently replace an unknown observation time with receipt time. Include units in the schema rather than relying on memory. Decide whether missing, invalid, saturated, stale, and estimated values are distinct states. Floating-point NaN and infinities are not valid standard JSON.

3.7 A Small Validated Web Service

The following executable Flask example uses a deliberately small, single-temperature contract to demonstrate validation and separate health endpoints [42, 41]. Unlike the richer representation above, this contract uses the field temperature_C with a fixed Celsius unit. Requests must contain exactly four fields: schema_version, device_id, temperature_C, and observed_at. The timestamp format is UTC with seconds and an optional fractional part, ending in Z.

This local teaching service retains only the most recently accepted request in one process. It has no durable storage, authentication, or deduplication, and loses its sample on restart. Therefore a 201 response here means acceptance into memory, not durable ingestion. Multiple worker processes would have separate copies of this state. The laboratory extends these limitations with persistent state, client-generated identifiers, and the management boundary defined later in the chapter.

Listing 3.3. A small validated measurement API

1from datetime import datetime, timezone2from threading import Lock3from flask import Flask, jsonify, request4from werkzeug.exceptions import HTTPException5import math6import re7import uuid8 9app = Flask(__name__)10app.config["MAX_CONTENT_LENGTH"] = 16 * 102411latest = None12state_lock = Lock()13FIELDS = {"schema_version", "device_id", "temperature_C", "observed_at"}14UTC_TIME = re.compile(15    r"[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:"16    r"[0-9]{2}(?:\.[0-9]{1,6})?Z"17)18 19def problem(status, code, detail):20    return jsonify({"error": code, "detail": detail}), status21 22@app.errorhandler(HTTPException)23def http_error(error):24    response = error.get_response()  # Preserve status and headers.25    response.data = app.json.dumps({26        "error": "http_" + str(error.code),27        "detail": error.description,28    })29    response.content_type = "application/json"30    return response31 32@app.get("/health/live")33def live():34    return {"status": "alive"}, 20035 36@app.get("/health/ready")37def ready():38    # This in-memory example has no external dependencies.39    return {"status": "ready", "storage": "volatile"}, 20040 41@app.get("/api/v1/measurements/latest")42def get_latest():43    with state_lock:44        sample = latest45        if sample is None:46            return problem(404, "no_measurement", "No sample is available")47        return jsonify(sample), 20048 49@app.post("/api/v1/measurements")50def add_measurement():51    global latest52    if not request.is_json:53        return problem(415, "json_required", "Use application/json")54    body = request.get_json()  # Malformed JSON becomes HTTP 400.55    if not isinstance(body, dict):56        return problem(422, "object_required", "JSON object required")57    if set(body) != FIELDS:58        return problem(422, "invalid_fields", "Use the four schema fields")59    if type(body["schema_version"]) is not int or body["schema_version"] != 1:60        return problem(422, "invalid_schema", "schema_version must be 1")61 62    device_id = body["device_id"]63    value = body["temperature_C"]64    observed_at = body["observed_at"]65    if not isinstance(device_id, str) or not re.fullmatch(66        r"[A-Za-z0-9][A-Za-z0-9_-]{0,63}", device_id67    ):68        return problem(422, "invalid_device_id", "Invalid device identifier")69    # bool is a subclass of int: exclude it and numeric strings.70    if type(value) not in (int, float) or not -50 &lt;= value &lt;= 100:71        return problem(422, "invalid_value", "Expected -50 to 100 degrees C")72    if not math.isfinite(value):73        return problem(422, "invalid_value", "Temperature must be finite")74    if not isinstance(observed_at, str) or not UTC_TIME.fullmatch(observed_at):75        return problem(422, "invalid_time", "Expected a UTC timestamp ending Z")76    try:77        observed = datetime.fromisoformat(observed_at[:-1] + "+00:00")78    except ValueError:79        return problem(422, "invalid_time", "Invalid calendar date or time")80 81    sample = {82        "schema_version": 1,83        "measurement_id": str(uuid.uuid4()),84        "device_id": device_id,85        "temperature_C": float(value),86        "observed_at": observed.isoformat().replace("+00:00", "Z"),87        "received_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),88    }89    with state_lock:90        latest = sample91    return jsonify(sample), 201

Save the listing as measurement_api.py, install Flask in the project virtual environment, and run flask --app measurement_api run --host 127.0.0.1 for local development. A valid POST body is shown below; test it against /api/v1/measurements.

Listing 3.4. Request body for the teaching API

1{2  "schema_version": 1,3  "device_id": "device-017",4  "temperature_C": 22.7,5  "observed_at": "2026-09-18T12:04:31.420Z"6}

The timestamp parser checks syntax and calendar validity; it does not prove clock synchronization or data freshness. Define an acceptable age and clock-skew policy for deployment. The receipt-time field is generated by the server, and GET returns the last accepted request rather than the greatest observation timestamp. A deployed version also needs a production application server and appropriate transport security.

Validation must occur at the trust boundary. Range validation alone does not prove physical plausibility; a value can be within range but stale, stuck, or inconsistent with other sensors. Authentication identifies a principal; authorization decides what that principal may do.

3.8 Timeouts, Deadlines, Retries, and Backoff

A timeout bounds how long one activity waits. Separate connection, read, write, and overall operation timeouts where the library permits. A deadline is the remaining end-to-end time budget and should propagate across dependent calls. If three services each independently wait ten seconds, a supposedly ten-second user operation can take far longer.

Retry only when failure may be transient and the operation can be repeated safely. Use a finite attempt limit, exponential backoff, and random jitter so many clients do not retry simultaneously.

One common full-jitter schedule is

d_n ∼ U(0, min(d_max, d₀2ⁿ)), n = 0, 1, 2, …

where d₀ sets the initial delay window, dₘₐₓ caps the actual delay, and U(a, b) denotes a uniform random draw between a and b. The equation does not determine policy: the application must decide which errors are retryable and when the result is no longer useful.

Do not retry permanent validation failures such as malformed JSON or unauthorized access. Respect server guidance such as Retry-After where appropriate. A circuit breaker can temporarily stop requests to a persistently failing dependency, but it must not become invisible permanent disconnection.

Block 314:00–14:50content

Block 3: 14:00–14:50 · Reliability, Offline Operation, and Observability

Embedded Linux — Chapter 3, Part 3

3.9 Idempotency, Deduplication, Ordering, and Reconciliation

An operation is idempotent when repeating it has the same intended effect as performing it once. Reading a resource is naturally idempotent. “Set target temperature to 22” can be idempotent; “increase target by 1” is not.

For a non-idempotent submission, the client can generate a unique operation or idempotency key. The server stores the outcome associated with that key and returns the same logical result when the request is repeated. The retention period and scope of keys must be defined.

Measurement identifiers and sequence numbers help detect duplicates and gaps. Sequence numbers eventually wrap and may reset after device replacement, so interpret them together with device identity and boot or session identity. Timestamps alone are insufficient because clocks can be wrong or repeated.

After reconnection, reconciliation decides what to upload, discard, merge, or reject. Define ordering, conflict policy, maximum backlog age, storage limit, and the response when the queue becomes full. Silent deletion is rarely acceptable engineering behavior.

3.10 Offline Operation and Store-and-Forward

Embedded networks are intermittent. A store-and-forward gateway accepts local measurements, commits them to durable storage, and later transmits them. Delivery states may include created, durable, attempted, acknowledged, rejected, and expired.

Acknowledgment should correspond to a defined durability point. A TCP send completing does not mean the application server stored the record. Conversely, an application acknowledgment can be lost after storage, leading the sender to retry. This is why deduplication is required even with reliable transport.

“Exactly once” is usually an end-to-end application property built from identifiers, durable state, and idempotent processing, not a simple network switch. Many systems realistically implement at-least-once delivery plus deduplication.

3.11 Observability: Logs, Metrics, and Traces

Observability is the ability to infer internal behavior from emitted evidence. The three common signals answer different questions:

Signal	Strength	Embedded example
Logs	Contextual discrete events	Reconnection failed with endpoint and error category
Metrics	Efficient numerical time series	Queue depth, stale-data age, restart count, request latency
Traces	Path of one operation across components	Measurement identifier followed from gateway to API and database

Structured logs should include timestamp, severity, component, event, device or operation identifier, and safe context. Do not log passwords, private keys, bearer tokens, session cookies, or unnecessary personal data. Use consistent machine-readable fields instead of forcing dashboards to parse creative prose.

Useful metrics include acquisition success ratio, age of last valid sample, queue depth and oldest-item age, reconnect count, request count by status class, latency percentiles, dropped messages, CPU, memory, storage, temperature, restart count, and watchdog events. Metric labels must have bounded cardinality; using a unique measurement ID as a metric label can exhaust monitoring storage.

Figure 3.3. A useful operational dashboard emphasizes service outcomes, freshness, saturation, and actionable failure rather than decorative activity.

3.12 Health Endpoints, Dashboards, and Alerts

A liveness endpoint should answer whether the process can make progress; it should not fail merely because every remote dependency is temporarily unavailable, or an orchestrator may restart a healthy process repeatedly. Readiness describes whether useful requests should be routed to the instance. Application health includes domain evidence such as freshness of the latest valid sample.

Health endpoints must be cheap, bounded, and protected from revealing secrets or internal topology. A check that performs an unbounded chain of downstream calls can amplify failure.

Dashboards support investigation; alerts request action. Every alert should have an owner, severity, meaningful condition, delay or persistence rule, and response guidance. Alert on sustained stale data or exhausted buffering, not every short-lived packet loss. Measure recovery time and false-alert rate.

3.13 Secure Remote Management and Trust Boundaries

Remote management can change software, configuration, credentials, and physical behavior. It therefore deserves stronger protection than a public read-only status page.

A trust-boundary analysis should identify:

operators, devices, services, and automated update authorities;
credentials and cryptographic keys held by each principal;
reachable interfaces, ports, networks, and administrative actions;
authentication, authorization, encryption, logging, and rate limits;
update signing and verification keys;
recovery when credentials are lost, stolen, expired, or rotated; and
safe behavior when the management channel disappears or is suspected compromised.

Use defense in depth: network segmentation or VPN, encrypted protocols, unique device credentials, least privilege, restricted administrative accounts, supported software, audit logs, and physical access protection. Encryption without authenticating the intended peer can protect a connection to the wrong system.

Remote updates should verify artifact authenticity and integrity, check compatibility and available resources, install without destroying the known-good version, activate atomically where possible, health-check the new release, and roll back automatically or operationally. Update signing keys should not reside on the deployed device as signing keys; devices require verification material.

Engineering caution

Do not expose an unauthenticated development server, debug endpoint, or device-control API directly to the Internet. Framework development servers are designed for development convenience, not production isolation, transport security, rate limiting, or sustained hostile traffic.

Block 415:00–15:50content

Block 4: 15:00–15:50 · Secure Remote Management, Laboratory, and Assessment

Embedded Linux — Chapter 3 Part 4

3.14 Guided Laboratory: Build an Observable Web Service

Laboratory objective

Implement a versioned HTTP measurement resource on the embedded Linux target. Demonstrate validation, bounded waiting, structured evidence, offline or dependency-failure behavior, and secure management assumptions. The result must remain diagnosable when a component fails.

Part A—contract. Define the resource paths, methods, success responses, errors, maximum body size, schema version, device and measurement identifiers, timestamp format, units, range rules, and quality states. Provide at least one valid and three invalid examples.

Part B—implementation. Implement POST and latest-measurement GET operations. Add stable error objects and prevent stack traces from reaching clients. Run the application as the restricted systemd service created in Module 2.

Part C—health. Implement separate liveness and readiness endpoints. Define a domain metric for age of the latest valid sample. Explain which dependencies each endpoint checks and why.

Part D—observability. Produce structured logs with a request or correlation identifier. Expose or record at least request count by result, request latency, queue depth, and stale-data age. Bound log and metric growth.

Part E—failure injection. Test malformed JSON, oversized body, out-of-range value, duplicate measurement ID, delayed dependency, refused connection, lost response, stopped service, full queue, and recovery. Record expected and observed behavior.

Part F—remote management boundary. Draw the management and data paths. Mark identities, credentials, encrypted links, reachable ports, authorization decisions, update authority, and recovery access. Do not include real secrets in the submission.

Portfolio evidence

Include the API and data contracts, network and trust-boundary diagram, source revision, service configuration, timeout and retry table, idempotency mechanism, structured log examples, metric definitions, dashboard or plots, failure-test matrix, and measured recovery. Explain what a remote operator can determine without physical access and what remains unknowable.

3.15 Self-Assessment and Job-Interview Preparation

These questions are designed for Module 3 and for future interviews in embedded Linux, networking, IoT, robotics, cloud, and site reliability engineering. Strong answers describe uncertainty and trade-offs rather than promising perfect networks. Attempt each question before consulting the answers.

Questions

Q1.

Why can a client timeout occur even when the server completes and stores the operation?

Q2.

Define partial failure and give an embedded-system example.

Q3.

Why should device identity be independent of IP address? Does a stable identifier prove authenticity?

Q4.

A device responds to ping but its HTTPS API is unavailable. What has ping proved, and what should be tested next?

Q5.

Explain when GET, POST, PUT, PATCH, and DELETE are appropriate.

Q6.

Distinguish HTTP 400, 401, 403, 404, 409, 422, 429, and 503.

Q7.

What fields are needed for a useful measurement contract beyond a numerical value?

Q8.

Why should observation time and server receipt time be separate?

Q9.

Explain connection timeout, read timeout, and end-to-end deadline.

Q10.

Why are exponential backoff and jitter used together?

Q11.

When is retrying a POST unsafe, and how can an idempotency key help?

Q12.

Compare at-most-once and at-least-once delivery. How can at-least-once delivery support correct results?

Q13.

What information can a sequence number provide, and what ambiguity remains after restart or wraparound?

Q14.

Distinguish logs, metrics, and traces using one sensor-data example for each.

Q15.

Why is “the process is running” an inadequate health check?

Q16.

What is dangerous about using measurement IDs or user IDs as metric labels?

Q17.

Design degraded behavior for a monitoring gateway during a six-hour Internet outage.

Q18.

What controls belong at a remote-management boundary?

Q19.

Describe a safe remote-update workflow and state what rollback must preserve.

Q20.

A dashboard shows no new data. Give a layered investigation that avoids assuming the sensor is faulty.

Answers and discussion

A1. The request may reach the server and commit successfully, while processing exceeds the
client’s wait or the response is lost. The client sees uncertainty, not proof of failure, so blind
repetition may duplicate work.

A2. Partial failure means one component or communication path fails while others continue. For
example, a gateway continues sampling and storing locally while DNS or the cloud API is
unreachable.

A3. Addresses change with networks and DHCP, while the logical device should persist. A stable
label supports continuity but can be copied; authenticity requires protected credentials and
verified proof.

A4. Ping indicates that an ICMP exchange succeeded at that time. Next test name resolution if
a name is used, route and TCP connection to port 443, server listening state, firewall, TLS
negotiation, HTTP response, authentication, and application dependencies.

A5. GET retrieves without intended state change; POST submits or creates subordinate resources;
PUT creates or replaces a known resource; PATCH applies a defined partial modification;
DELETE requests removal. Actual API semantics must remain consistent with these expec-
tations.

A6. 400 indicates a request rejected for a perceived client error, such as malformed syntax; 401
failed or missing authentication; 403 refusal to fulfill the request, whether or not authentication
succeeded; 404 absent resource; 409 state conflict; 422 syntactically valid but semantically
invalid content; 429 rate limit; 503 temporary service unavailability.

A7. Include schema version, device and measurement identity, observation timestamp, unit, value
type and range, quality/status, sequence or session information where useful, and an explicit
missing-value policy.

A8. Observation time describes the physical event; receipt time describes arrival. Network outages,
buffering, and clock faults make them different. Replacing one with the other corrupts latency
and temporal interpretation.

A9. Connection timeout bounds establishment of transport; read timeout bounds waiting for
response data; an end-to-end deadline limits the complete operation across retries and
dependencies. Per-call timeouts must fit inside the remaining deadline.

A10. Exponential backoff reduces load during sustained failure. Jitter prevents synchronized clients
from retrying together and causing a new overload wave.

A11. POST is unsafe to retry when repeating it can create another resource or repeat physical
action. A client-generated key lets the server recognize the same logical operation and return
its stored outcome rather than perform it again.

A12. At-most-once may lose work but avoids repetition; at-least-once retries and may duplicate.
Unique identifiers, durable deduplication, and idempotent downstream processing can make
repeated delivery produce one intended effect.

A13. It reveals local ordering and possible gaps or duplicates. Wraparound, device reset, restored
backup, or replacement can reuse values, so combine it with persistent device and boot/session
identity.

A14. A log records “upload failed: timeout” with context; a metric counts failures and latency over
time; a trace follows measurement 1842 through gateway, API, and storage. They complement
rather than replace one another.

A15. The process can be deadlocked, disconnected, reading stale input, unable to write, or
generating invalid data. Health must include progress, dependency state where appropriate,
and application outcome or freshness.

A16. Each unique label combination creates a time series. Unbounded identifiers cause cardinality
explosion, consuming memory, storage, and query capacity. Put high-cardinality identifiers
in logs or traces instead.

A17. Continue timestamped local acquisition, commit to a bounded durable queue, expose offline
and backlog status, apply an explicit full-queue policy, reconnect with backoff and jitter,
upload oldest valid records with IDs, deduplicate server-side, and verify backlog convergence
after recovery.

A18. Define operator and device identities, authentication, authorization by action, encrypted
channels, network reachability, secret storage and rotation, audit logging, rate limits, update
verification, physical recovery, and safe behavior if the channel is unavailable or compromised.

A19. Verify signed artifact, compatibility, resources, and configuration; retain the known-good
release; install separately; activate atomically; run bounded health checks; and revert on
failure. Rollback must preserve compatible executable, service definition, configuration,
credentials, and data schema or snapshot.

A20. Check dashboard time range and query first, then server ingestion and storage, API errors
and latency, authentication, transport and DNS, gateway queue and logs, gateway service
health, local measurement freshness, device permissions and interface, and finally physical
sensor behavior. Correlate identifiers and timestamps across layers.

Chapter summary

Distributed embedded systems operate under uncertainty. Reliable designs separate identity from location, use versioned resource and data contracts, bound waiting, retry only when safe, detect duplicates, buffer disconnections, and reconcile after recovery. Logs, metrics, traces, and health checks expose failures. Secure remote management protects observation, configuration, and updates. These principles support MQTT, LoRaWAN, ROS 2, automation, and edge computing.

Summary

Distributed embedded systems operate under uncertainty. Reliable designs separate identity from location, reason about partial failure, handle latency and duplication deliberately, and provide evidence through logs, metrics, traces, health checks, and recovery tests. This module develops these principles through HTTP services, measurement contracts, retries and backoff, idempotency, offline operation, observability, and secure remote management.

Course outline

What we will learn

From Linux foundations to physical interfaces, networking, security and a complete embedded system.

Linux foundationsKernel, users, processes and file systems

Command lineShells, scripting and automation

Embedded hardwareBoot, storage and device drivers

Physical I/OGPIO, sensors and actuators

Device busesUART, I²C and SPI

NetworkingTCP/IP, SSH and remote management

Distributed systemsMessage passing and cooperating nodes

Engineering practiceGit, testing, logging and deployment

SecurityLeast privilege, updates and secrets

Teaching method

Short introductions are followed by laboratory activities. Work is normally completed in teams of three. Each team develops an embedded Linux system; the examination portfolio is individual.

Project ideas

Applications you could build

Each project should begin with one reliable end-to-end function.

Automatic Raspberry Pi-based GPS trackerSample examination report available in PDF format

Environmental monitoring stationValidated sensing, local storage and dashboard

Energy monitoring systemMeasure, log and visualize equipment power use

Greenhouse automationClimate sensing, actuation and remote supervision

Field-robot telemetryLocation, operating state, measurements and faults

LoRaWAN sensor networkField nodes, TTN integration and private cloud

Intelligent irrigationSensing, decisions, valves and safe failure handling

Multi-node IoT gatewayProtocol translation, buffering and secure forwarding

ROS 2 communication systemDistributed nodes, QoS and service integration

Edge-AI monitoring systemLocal inference, measured performance and safe fallback

A complete project should demonstrate

NeedA clear user and purpose

Physical interfaceA sensor, actuator or device

Linux integrationPrograms, scripts or services

CommunicationA justified data-exchange method

ReliabilityLogging and failure handling

EvidenceTests, results and reflection

Contractor and client roles

Project groups

Each group develops a project for one client group and evaluates the project received from another contractor group.

Contractor

Heros

ProjectProject 01

Delivers to: Client not assigned
Receives from: Contractor not assigned

Fridays · four 50-minute slots · Odense

Fall 2026 lecture plan

The SDU timetable remains authoritative for dates, times and rooms.

No.	Date	Lecture topic
01	04 Sep 2026	Linux fundamentals and development environment
02	11 Sep 2026	Embedded Linux Platforms, Raspberry Pi, Services, and Automation
03	18 Sep 2026	Distributed Networking, Web Services, Monitoring, and Remote Management
04	25 Sep 2026	Sensors, Actuators, GPIO, and Serial Interfaces
05	02 Oct 2026	Microcontrollers and Distributed Embedded Nodes
06	09 Oct 2026	Wireless Acquisition, LoRa Peer-to-Peer, and MQTT
07	23 Oct 2026	LoRaWAN, TTN, Gateways, Tracking, and Private-Cloud Integration
08	06 Nov 2026	ROS 2: Robot Integration, Simulation, and Deployment
09	13 Nov 2026	Home Assistant, Edge AI, Local LLMs, and Intelligent Systems
10	20 Nov 2026	Project Integration, Security, and Verification
11	27 Nov 2026	Implementation, Debugging, and Deployment
12	04 Dec 2026	Demonstrations and the Individual Portfolio

Project progression

Project ideas and proposals begin in Module 1 and evolve throughout the course. Module 10 is the formal integration, architecture and verification review; Modules 11–12 complete implementation, demonstration and individual portfolio evidence.

Lecture 03 · participation

Student questions

Questions remain hidden until the student is called. Model answers are revealed separately.

Instructor classroom view

The roster, answers, attendance and participation records are available only to the authorized primary owner after sign-in.

Instructor sign in

Definitions

Essential course vocabulary

Plain-language definitions for the technical terms used throughout the course.

Term	Definition	Example
Actuator	A device that produces a physical action.	Motor, valve, relay or fan
Boot	The sequence that starts the hardware, kernel, services and applications.	Starting an embedded computer
CLI	A text interface that accepts commands.	Windows Terminal or a Linux terminal
Command	An instruction interpreted by a shell.	pwd, ls or Get-Location
Device driver	Software through which the operating system communicates with hardware.	GPIO or camera driver
Directory	A named container for files and other directories.	/home/student/eml
Distribution	A complete operating system built around the Linux kernel.	Ubuntu or Debian
Embedded system	A computer integrated into a larger product for a defined purpose.	Greenhouse controller
File system	The structure used to name, store and retrieve files.	ext4
GPIO	A configurable digital input or output pin.	Reading a switch
I²C	A two-signal serial bus for peripherals.	Environmental sensor
Kernel	The privileged core managing memory, processors, devices and processes.	Linux kernel
MQTT	A lightweight publish/subscribe messaging protocol.	Sending sensor data
Process	A running instance of a program.	A data logger
PWM	Rapid switching used to control average power or a signal duty cycle.	Motor speed
Root	The privileged Linux user or the top directory, depending on context.	Administrative access
Script	A file containing commands that automate work.	A backup script
Sensor	A device that converts a physical quantity into data.	Temperature sensor
Service	A background program providing an ongoing function.	SSH server
Shell	A program that interprets commands and starts programs.	Bash or PowerShell
SPI	A synchronous bus commonly used for fast peripherals.	Display or converter
SSH	Encrypted remote command-line access.	Managing a Raspberry Pi
TCP/IP	The principal protocol family used across computer networks.	Device-to-server communication
UART	An asynchronous serial interface.	Debug console
User space	The environment where ordinary programs run outside the kernel.	Application software
Version control	A system that records and manages file changes.	Git

Official examination information · EKA T550077102

Individual portfolio examination

Form

Portfolio

Individual presentation portfolio submitted at semester end.

Grade

7-point scale

No second examiner.

Weighting

No split published

No separate written-exam or coursework percentages are stated.

Re-examination

Same form

Same form as the ordinary examination.

Team project and individual assessment

The team project creates practical evidence, but each student is assessed individually through the presentation portfolio. Classroom participation points support engagement and do not form part of the official grade.