Postlad/Guides/Store and forward after an outage

Store and forward: send buffered readings after an outage

A logger that only sends live readings loses everything the network was down for. The fix is old and simple: keep the reading locally with the time it was taken, and send the backlog when the connection returns. Postlad's batch endpoint accepts up to 100 timestamped readings in one request, so the gap in the chart fills in with the real times rather than with the moment the Wi-Fi came back.

Tutorial·9 min read·Last checked August 2026

The endpoint

A single live reading goes to /u. A list of buffered readings goes to /u/batch, as JSON:

POST https://api.postlad.com/u/batch
Content-Type: application/json

{
  "k": "pk_live_YOUR_KEY",
  "points": [
    {"t": 1754604000, "f1": 21.9, "f2": 55.1},
    {"t": 1754604030, "f1": 21.8, "f2": 55.4},
    {"t": 1754604060, "f1": 21.8, "f2": 55.6}
  ]
}

The reply is JSON, and it counts what happened rather than assuring you generally:

{"ok":true,"stored":3,"updated":0,"month_points":4118,"stream":"greenhouse","first_ts":1754604000000,"last_ts":1754604060000}

stored is readings that did not exist before. updated is readings whose timestamp was already there and which were filled in rather than duplicated — the number that makes a nervous device's second attempt safe.

The rules that matter

Four batches a minute on the free plan is 400 buffered readings a minute. A day of readings taken every 30 seconds is 2,880 points, so that backlog clears in about eight minutes. The monthly reading allowance is the limit that actually binds: a batched reading counts exactly the same as a live one, so 100,000 a month on Free means 100,000 however they arrive.

The device has to know the time

A timestamp you invented is worse than no timestamp. Send t only when the device genuinely knows the clock:

• While online with no buffer to send, leave t off entirely. The server timestamps the reading on arrival, which is accurate to the second and free of clock bugs.
• ESP32 / ESP8266: call configTime() once after joining Wi-Fi, then time(nullptr) gives unix seconds.
• Pico W or other MicroPython boards: ntptime.settime() at boot, then time.time() with the epoch offset your port uses.
• Linux: it is already right; int(time.time()).
• No clock at all until reconnection? Record each reading's millis() value instead, and convert on flush: t = now − (millis_now − millis_then) / 1000. It is an offset from a known point rather than a guess.

A Python logger that survives the gap

The pattern in about forty lines: append every reading to a file, then try to drain the file. If the network is down the append still happens and nothing is lost.

import json, os, time, requests

WRITE_KEY = os.environ.get("POSTLAD_KEY", "pk_live_YOUR_KEY")
BATCH_URL = "https://api.postlad.com/u/batch"
SPOOL     = "spool.jsonl"
MAX_POINTS = 100          # the batch endpoint's ceiling


def record(f1, f2):
    """Never skipped, whatever the network is doing."""
    with open(SPOOL, "a") as fh:
        fh.write(json.dumps({"t": int(time.time()), "f1": f1, "f2": f2}) + "\n")


def load():
    if not os.path.exists(SPOOL):
        return []
    with open(SPOOL) as fh:
        return [json.loads(line) for line in fh if line.strip()]


def save(points):
    tmp = SPOOL + ".tmp"
    with open(tmp, "w") as fh:
        for p in points:
            fh.write(json.dumps(p) + "\n")
    os.replace(tmp, SPOOL)          # atomic: a crash mid-write cannot truncate the spool


def flush(session):
    points = load()
    while points:
        chunk = points[:MAX_POINTS]
        try:
            resp = session.post(BATCH_URL, json={"k": WRITE_KEY, "points": chunk}, timeout=15)
        except requests.RequestException as exc:
            print(f"offline, {len(points)} point(s) buffered: {exc}")
            return

        if resp.status_code == 200:
            points = points[len(chunk):]
            save(points)
            continue

        if resp.status_code == 207:
            # The batch straddled the monthly cap. The server stored a prefix and
            # named the index where it stopped: keep everything from there on.
            first_rejected = resp.json().get("first_rejected_index", 0)
            points = points[first_rejected:]
            save(points)
            print(f"quota reached, {len(points)} point(s) kept: {resp.text.strip()[:200]}")
            return

        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", "30"))
            print(f"rate limited, waiting {wait}s")
            time.sleep(wait)
            continue

        if resp.status_code >= 500:
            print(f"server error {resp.status_code}; keeping the buffer")
            return

        # Any other 4xx means this request is wrong and will stay wrong.
        print(f"refused {resp.status_code}: {resp.text.strip()[:200]}")
        return


if __name__ == "__main__":
    session = requests.Session()
    while True:
        record(round(21.0 + time.time() % 3, 2), 55.0)   # your sensor here
        flush(session)
        time.sleep(30)

Two details are doing the real work. The write to the spool happens before any network call, so a reading is never conditional on connectivity. And the spool is only shortened by exactly what the server said it stored — never by what we hoped it stored.

For an unattended Linux logger, the durable version of this — SQLite spool, systemd unit, restart policy — is written out in full on Raspberry Pi data logger online.

The same thing on an ESP32

A microcontroller usually buffers in RAM, which means the buffer is bounded and a reset costs it. That is an acceptable trade for most projects; an SD card or LittleFS is the upgrade when it is not.

#include <WiFi.h>
#include <HTTPClient.h>
#include <time.h>

const char* WRITE_KEY = "pk_live_YOUR_KEY";
const char* BATCH_URL = "http://api.postlad.com/u/batch";
const size_t BUFFER_MAX = 60;          // readings held in RAM; the endpoint takes 100 per request

struct Reading { time_t t; float f1; };
Reading buffer[BUFFER_MAX];
size_t count = 0;

void remember(time_t t, float f1) {
  if (count == BUFFER_MAX) {           // full: drop the oldest, keep logging
    memmove(buffer, buffer + 1, sizeof(Reading) * (BUFFER_MAX - 1));
    count--;
  }
  buffer[count++] = Reading{ t, f1 };
}

bool flushBuffer() {
  if (count == 0) return true;
  if (WiFi.status() != WL_CONNECTED) return false;

  String body = String("{\"k\":\"") + WRITE_KEY + "\",\"points\":[";
  for (size_t i = 0; i < count; i++) {
    if (i) body += ",";
    body += "{\"t\":" + String((long)buffer[i].t) + ",\"f1\":" + String(buffer[i].f1, 2) + "}";
  }
  body += "]}";

  HTTPClient http;
  http.begin(BATCH_URL);
  http.addHeader("Content-Type", "application/json");
  int status = http.POST(body);
  String reply = http.getString();
  http.end();

  Serial.printf("flush %u -> HTTP %d %s\n", (unsigned)count, status, reply.c_str());
  if (status == 200) { count = 0; return true; }
  return false;                        // keep the buffer and try again next loop
}

Call remember(time(nullptr), temperature) every cycle and flushBuffer() straight after. While the link is up the buffer holds one reading and empties immediately; while it is down the buffer fills; when the link returns the whole backlog goes in one request.

The sketch above clears the buffer only on a 200. That is the conservative choice and it is the right default — the cost of resending a reading is nothing, because a timestamp that already exists is updated rather than duplicated.

Pair it with a silence alert

Buffering fixes the data. It does not tell you the device stopped, and a device that has been buffering for three weeks because its power supply failed has not been buffering at all. The two halves belong together.

On the stream's page, add an alert rule of the kind The stream goes quiet and give it a window — anything from 2 minutes to 7 days, with 30 minutes as the default. When no reading has arrived for that long, you get an email; when the stream reports again, you get the matching recovery notice.

• The free plan includes two alert rules, delivered by email. Rules that watch a value — above, below, outside a range — are part of Hobby and Lab, because they are checked against every reading as it arrives.
• Nothing is on by default. No rule exists until you create one, and Postlad never emails about your data unless you asked it to.
• Detection resolution is about a minute, because the check runs on a once-a-minute sweep.
• At least 30 minutes between emails from one rule on Free — 5 minutes on Hobby, 1 minute on Lab.

The two features are designed to behave sensibly together:

• Old readings are alert-inert. Anything timestamped more than about 15 minutes from now is not evaluated by an alert rule at all, so replaying a day of buffer cannot produce a day of email. Genuinely recent readings in the same batch are still checked, which is the behaviour you want.
• A late reading never makes a live stream look dead. The stream's last-seen time only ever moves forward, so posting an hour-old reading does not drag it backwards into a silence alert.
• The recovery notice follows the newest reading in what you send. A device that returns and flushes a backlog ending with a fresh sample is reporting again, and is told so.

What not to do

• Do not send zeros for missing samples. Zero degrees is a valid-looking measurement. A gap is honest and easy to diagnose; a fabricated zero survives into the chart, the CSV and eventually into a conclusion.
• Do not retry a 400. It means the request itself is wrong and it will be wrong forever. Log the sentence — it names the exact problem — and move on.
• Do honour Retry-After. A 429 says the reading was not stored and when to send it again. Hammering the endpoint only lengthens the outage.
• Do not backfill history through the batch endpoint. It is for a recent gap. A year of rescued CSV goes through the importer, which has its own allowance and does not spend your monthly quota.
• Do not let the buffer grow without a ceiling on a device that cannot afford it. Decide in advance whether an old reading or a new one gets dropped when memory runs out, rather than finding out during a reset.

Troubleshooting

FAQ

Does a batched reading cost more than a live one?

No. One reading is one reading against the monthly allowance however it arrived. Batching is cheaper for us to store, which is why the batch limit is generous relative to the single-write interval.

What if I send the same buffer twice?

Nothing bad. A reading whose timestamp already exists is updated in place, and the reply reports it under updated instead of stored. When in doubt, resend — that is the safer error.

How far back can a batch reach?

7 days. Older readings belong to the CSV importer, which exists precisely so that a long backlog does not have to squeeze through the device path. The refusal sentence tells you how many days out the point was.

Can I mix live writes and batches?

Yes, and most loggers should. Send live readings to /u while the link is up; use /u/batch only to clear a backlog. The two paths share the same key, stream, quota and rules.

What does the chart do with a filled gap?

It redraws with the readings at their real times. Within the last 30 days on Free the readings are at full detail; once a period ages past that it is summarised as an hourly minimum, maximum and average, kept for 12 months, and the chart draws that envelope rather than an average alone.

Create a stream and try the gap

The honest test takes five minutes: start the logger, pull the network cable, put it back, and watch the chart fill in behind itself.

Create your free account →

Related guides

• Raspberry Pi data logger online: the durable version — SQLite spool, systemd unit, replay on reconnect.
• Send ESP32 data to the cloud: the live sender this page adds a buffer to.
• Point your ThingSpeak sketch at Postlad: the /update alias, and what changes in the reply.
• Move a CSV history across: for backlogs older than the batch window.
• ESP32 water tank level monitor: a remote install where the network is the unreliable part.

Sources and verification

• Every limit, status code and reply shape on this page was read from the shipped ingest worker and the tests that pin it: the batch parser, the point ceiling, the timestamp window, the partial-quota response and the alert engine's backfill guard.
• The Python and Arduino code here is written for this guide and reviewed rather than bench-run on every board; the spool pattern it follows is the one running in the Raspberry Pi logger.
• Error sentences are generated from one catalogue, so the wording your device receives may be more specific than the summary in the table above. The full list is at app.postlad.com/docs/errors.