Postlad/Guides

Raspberry Pi data logger, online

Two lines of Python put a reading on the internet. The rest of this page is about the part that actually matters for a logger you leave running: what happens when the WiFi drops at 3am, the router reboots, or the endpoint has a bad five minutes. Below is the five-line version to prove the path works, then a complete systemd service that buffers readings to disk and sends them in batches when the network comes back.

Tutorial·reviewed and dry-run against the live endpoint; Pi 3 bench pass still pending

The five-line version

import requests

requests.get(
    "https://api.postlad.com/u",
    params={"k": "YOUR_KEY", "f1": 23.5},
    timeout=10,
)

That is a complete data logger, in the sense that it puts a number on the internet. Run it in a cron job every minute and you have a working, if fragile, logger.

A slightly more honest version that reads something real — the Pi's own CPU temperature, so you need no wiring at all to try this:

#!/usr/bin/env python3
import requests

with open("/sys/class/thermal/thermal_zone0/temp") as f:
    cpu_temp = int(f.read().strip()) / 1000.0        # the file is in millidegrees C

r = requests.get(
    "https://api.postlad.com/u",
    params={"k": "YOUR_KEY", "f1": round(cpu_temp, 2)},
    timeout=10,
)
print(r.status_code, r.text.strip())

Save it as log.py, chmod +x log.py, run it. A stored point answers 200 with a plain-text ok and the number of fields it took. If requests is missing:

sudo apt install python3-requests

Use apt rather than pip install for a system-wide script. Recent Raspberry Pi OS releases mark the system Python as externally managed, so a bare pip install refuses to run and tells you to use a virtual environment — which the full service below does properly.

Why the five-line version is not enough

Leave that running for a week on a shelf and you will find the holes:

• The WiFi drops. requests.get raises, the script dies, cron logs it to a mail spool nobody reads, and you lose every reading until you notice.
• The endpoint has a bad minute. Same outcome, for a 30-second problem.
• The gap is invisible. When you finally look at the chart, there is a hole and no way to know whether the sensor failed, the Pi failed, or the network failed.
• Timestamps are wrong after a gap. Even if you retry, a reading taken at 03:14 that arrives at 07:40 will be plotted at 07:40 unless you send the original time with it.

A proper logger buffers to disk, retries with the original timestamps, and runs under something that restarts it. On a Pi that means sqlite3 and systemd, both already installed.

The full version: a systemd service with offline buffering

Three files. The logger writes every reading to a small SQLite spool first, then tries to flush the spool to the network in batches of up to 100 points. If the network is down, readings pile up in the spool and go out — with their original timestamps — when it comes back.

1. The logger

/opt/postlad/logger.py

#!/usr/bin/env python3
"""Read the Pi's CPU temperature and load average, send them to Postlad,
and buffer to disk whenever the network is unavailable."""

import os
import signal
import sqlite3
import sys
import time

import requests

ENDPOINT   = "https://api.postlad.com/u"        # single reading
BATCH_ENDPOINT = "https://api.postlad.com/u/batch"   # buffered readings, up to 100
WRITE_KEY  = os.environ.get("POSTLAD_KEY", "YOUR_KEY")
SPOOL_PATH = os.environ.get("POSTLAD_SPOOL", "/var/lib/postlad-logger/spool.db")

INTERVAL_SECONDS = 30        # the free plan's floor is 5 s, but its 100k points/month
                             # quota works out at one reading every 26 s over a month
BATCH_SIZE       = 100       # the batch endpoint accepts up to 100 points per request
MAX_SPOOL_ROWS   = 200_000   # ~70 days at 30 s; stops a long outage filling the SD card
HTTP_TIMEOUT     = 15

running = True


def stop(signum, frame):
    global running
    running = False
    print("stopping", flush=True)


signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)


def open_spool(path):
    os.makedirs(os.path.dirname(path), exist_ok=True)
    conn = sqlite3.connect(path)
    conn.execute("PRAGMA journal_mode=WAL")
    conn.execute(
        """CREATE TABLE IF NOT EXISTS spool (
               id INTEGER PRIMARY KEY AUTOINCREMENT,
               t  INTEGER NOT NULL,
               f1 REAL,
               f2 REAL)"""
    )
    conn.commit()
    return conn


def read_cpu_temperature():
    with open("/sys/class/thermal/thermal_zone0/temp") as f:
        return int(f.read().strip()) / 1000.0


def read_load():
    return os.getloadavg()[0]


def backlog(conn):
    return conn.execute("SELECT COUNT(*) FROM spool").fetchone()[0]


def record(conn, t, f1, f2):
    conn.execute("INSERT INTO spool (t, f1, f2) VALUES (?, ?, ?)", (t, f1, f2))
    conn.execute(
        "DELETE FROM spool WHERE id <= (SELECT MAX(id) - ? FROM spool)", (MAX_SPOOL_ROWS,)
    )
    conn.commit()


def drop(conn, rows):
    conn.executemany("DELETE FROM spool WHERE id = ?", [(r[0],) for r in rows])
    conn.commit()


def flush(conn, session):
    """Send buffered points oldest first. Returns True if the spool emptied."""
    while running:
        rows = conn.execute(
            "SELECT id, t, f1, f2 FROM spool ORDER BY id LIMIT ?", (BATCH_SIZE,)
        ).fetchall()
        if not rows:
            return True

        payload = {
            "k": WRITE_KEY,
            "points": [{"t": r[1], "f1": r[2], "f2": r[3]} for r in rows],
        }

        try:
            resp = session.post(BATCH_ENDPOINT, json=payload, timeout=HTTP_TIMEOUT)
        except requests.RequestException as exc:
            print(f"network unavailable, {backlog(conn)} point(s) buffered: {exc}", flush=True)
            return False

        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", "30"))
            print(f"rate limited, waiting {wait}s", flush=True)
            time.sleep(wait)
            continue

        if resp.status_code == 207:
            # The batch straddled the monthly quota: the server stored a prefix
            # and told us the index where it stopped. Delete exactly that prefix
            # and keep the rest — deleting the whole batch here would throw away
            # points the server has just said it did NOT store.
            first_rejected = resp.json().get("first_rejected_index", 0)
            print(f"quota reached; {first_rejected} of {len(rows)} stored, keeping the rest", flush=True)
            drop(conn, rows[:first_rejected])
            return False

        if resp.status_code >= 500:
            print(f"server error {resp.status_code}, will retry later", flush=True)
            return False

        if resp.status_code >= 400:
            # A 4xx that isn't 429 means the request itself is wrong - a bad key, a bad
            # field, a timestamp outside the accepted window. Retrying forever would spin,
            # so drop this batch and complain loudly in the journal.
            print(f"batch rejected ({resp.status_code}): {resp.text.strip()[:200]}", flush=True)
            drop(conn, rows)
            return False

        drop(conn, rows)
        print(f"sent {len(rows)} point(s); {backlog(conn)} still buffered", flush=True)

    return False


def main():
    if WRITE_KEY == "YOUR_KEY":
        sys.exit("POSTLAD_KEY is not set - see /etc/default/postlad-logger")

    conn = open_spool(SPOOL_PATH)
    session = requests.Session()
    next_reading = time.monotonic()

    while running:
        record(
            conn,
            int(time.time()),                      # server accepts device timestamps
            round(read_cpu_temperature(), 2),      # f1
            round(read_load(), 2),                 # f2
        )
        flush(conn, session)

        next_reading += INTERVAL_SECONDS
        deadline = time.monotonic() + max(0.0, next_reading - time.monotonic())
        while running and time.monotonic() < deadline:
            time.sleep(min(1.0, deadline - time.monotonic()))

    conn.close()


if __name__ == "__main__":
    main()

Two endpoints, not one. A single reading goes to /u; a list of buffered readings goes to /u/batch. Posting a points array to /u is rejected — /u takes one flat point and nothing nested — so if the journal shows bad_json complaining about "points", the URL is the bug.

Swapping in a real sensor. Replace read_cpu_temperature() and read_load() with whatever you have. A DS18B20 on 1-Wire reads out of /sys/bus/w1/devices/28-*/w1_slave; a BME280 over I²C uses adafruit-circuitpython-bme280. Nothing else in the file changes — the buffering does not care where the numbers came from.

Timestamps. Each row stores int(time.time()), the moment the reading was taken, and that value is sent with the point. A reading taken during an outage is plotted at the time it was taken, not the time it was delivered. This is the entire reason for the spool.

2. The key file

/etc/default/postlad-logger

POSTLAD_KEY=YOUR_KEY

sudo chmod 600 /etc/default/postlad-logger

The write key lets anyone who has it write to your stream. Keep it out of the script, out of your shell history and out of any repository.

3. The unit file

/etc/systemd/system/postlad-logger.service

[Unit]
Description=Postlad data logger
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=postlad
Group=postlad
EnvironmentFile=/etc/default/postlad-logger
WorkingDirectory=/opt/postlad
ExecStart=/opt/postlad/venv/bin/python /opt/postlad/logger.py
Restart=always
RestartSec=10
StateDirectory=postlad-logger
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

StateDirectory=postlad-logger makes systemd create /var/lib/postlad-logger with the right ownership, which is where the spool lives. Restart=always covers the crash case; After=network-online.target avoids a pointless failed attempt at boot, though the spool would handle that anyway.

Installing it

# a system user with no login and no home directory of its own
sudo useradd --system --home /opt/postlad --shell /usr/sbin/nologin postlad

sudo mkdir -p /opt/postlad
sudo cp logger.py /opt/postlad/logger.py

# a virtual environment, because recent Raspberry Pi OS marks the system Python
# as externally managed and refuses a system-wide pip install
sudo apt install -y python3-venv
sudo python3 -m venv /opt/postlad/venv
sudo /opt/postlad/venv/bin/pip install requests

sudo chown -R postlad:postlad /opt/postlad

sudo cp postlad-logger.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now postlad-logger

Watch it work:

journalctl -u postlad-logger -f

One line per reading, and one line per flush telling you how much is still buffered.

Testing the offline path

This is the part worth actually testing, because it is the part you are relying on at 3am:

# break the network
sudo ip link set wlan0 down

# watch readings pile up in the spool
sudo sqlite3 /var/lib/postlad-logger/spool.db "SELECT COUNT(*) FROM spool;"

# put it back
sudo ip link set wlan0 up

The journal should show the backlog draining in batches, and the chart should fill in the gap with points at their original times rather than a cliff of readings all stamped at the moment the network returned.

Troubleshooting

error: externally-managed-environment from pip. Recent Raspberry Pi OS follows PEP 668: the system Python is managed by apt and pip refuses to install into it. Either sudo apt install python3-requests for a quick script, or use the virtual environment as above for the service.

The service starts and immediately stops. systemctl status postlad-logger and then journalctl -u postlad-logger -n 50. The usual causes are a POSTLAD_KEY that is still YOUR_KEY (the script exits on purpose), a typo in the ExecStart path, or the postlad user not being able to read /opt/postlad.

Permission denied on the spool. StateDirectory= only creates the directory when systemd starts the service. If you ran the script by hand as your own user first, /var/lib/postlad-logger may be owned by the wrong account — sudo rm -rf /var/lib/postlad-logger and let systemd recreate it.

The clock is wrong after a power cut. A Pi has no battery-backed clock and gets the time from NTP at boot. If it boots with no network, early readings get timestamps from 1970 or from the last known time, and the endpoint will reject anything outside its accepted window. systemd-timesyncd handles this in the normal case; if your Pi lives somewhere with unreliable network, this is one of the few genuinely good reasons to fit an RTC module.

SD card wear. The spool writes a couple of small rows a minute, which is nothing, but SQLite in WAL mode plus journald plus a cheap card is a combination that has killed Pis before. If this is a long-term deployment, put the spool on a USB stick or an SSD and consider Storage=volatile in journald.conf.

High CPU on a Pi Zero. requests plus TLS is not free on the slowest boards. Raising INTERVAL_SECONDS is the first lever; the second is sending less often but in bigger batches, which the spool already supports — just call flush() every tenth loop instead of every loop.

Where it sends to

The endpoint above is live. A stream gets a write key; the device sends either a plain GET for a single reading or a batch POST for buffered ones; the stream has a live page with charts and a share link that a viewer opens with no account. The same chart embeds into a blog post or a wiki with one line of HTML, as a script tag or an iframe; free-tier embeds carry a small Postlad badge.

The free tier keeps every reading at full detail for 30 days, then an hourly minimum, maximum and average for 12 months — which is what makes a Pi that has been logging your loft temperature since last winter actually useful, and CSV export is there when you want the numbers out.

The buffering pattern above works against any HTTP endpoint, so nothing here is wasted if you use something else.

FAQ

How do I log Raspberry Pi sensor data online?

Read the sensor in Python, then requests.get or requests.post the value to a logging service. That is genuinely all of it for a first version — the two-line example at the top of this page works. The engineering is in the failure cases: buffering when the network is down, sending the original timestamps afterwards, and running under systemd so it restarts.

Should I use cron or systemd for a Raspberry Pi logger?

systemd, for anything you intend to leave running. Cron gives you a fresh process per run, which means no in-memory state, no easy backoff, and a restart story that consists of hoping. A systemd service with Restart=always handles crashes, gives you journalctl for free, and lets the process keep a spool and an HTTP session open between readings.

How do I handle gaps when my Raspberry Pi loses internet?

Write each reading to local storage first — SQLite is ideal and already installed — and only delete a row after the server has confirmed it. Store the time the reading was taken with each row and send that timestamp with the data, so the backfilled points land in the right place on the chart. The service above does exactly this, in batches of up to 100 points, and treats a partial acceptance as partial rather than throwing the whole batch away.

Can a Raspberry Pi Zero or Pi 3 handle this?

Comfortably. At a 30-second interval the script is idle almost all the time and the spool grows by a few kilobytes an hour. TLS handshakes are the most expensive part, which is why the code reuses a requests.Session() rather than opening a new connection per reading.

Do I need an RTC for accurate timestamps?

Usually not — the Pi syncs its clock over NTP at boot and keeps it there. You need an RTC only if the device regularly boots with no network and you are storing device-side timestamps, which is exactly the buffering case above. If your Pi is always online within a minute of booting, skip it.

Is this page tested on real hardware?

Not on a Pi yet. The code has been run against the live endpoint from a desktop Python — the backlog flushes and the replayed points land at their original times, which is the assertion this page lives or dies on — but a Raspberry Pi 3 bench pass on the real OS over real WiFi has not happened. It is the first page in our queue for that, and when it has been through it, this page will say so and name the OS version it was tested against. Until then it is carefully reviewed code, not proven code, and we would rather say so than let you find out.