Frame Receipt Format

The engine runs a frame through a fixed chain of stages. Each stage writes into its own folder of the run directory, <run>/<stage>/<product>. Early stages produce big working files: the decoded pixels, the color planes, every blob the detector found. Later stages shrink the frame down to the stars that were identified against the Gaia catalog and measured. The original file is never copied. The engine remembers it by path and fingerprint.

A receipt has three kinds of parts. Each has one job.

The schema registry, SCHEMAS.toml, lists every table and JSON product the engine writes and the version a reader must accept. Today it lists 32 Arrow tables and 19 JSON products. A test fails when a table's version in code disagrees with the registry.

The run receipt, by field

These are the real top-level fields of skycruncher.run_receipt version 1.4.0.

The run ID is a recipe, not a random number: sha256 over the build, the runtime switches, each input as role:fingerprint, and the stage list. Anyone holding the receipt can recompute it and ask for the same run again.

Normalized by level

Each fact lives at one level and is pointed to from the others. Part two carries this same rule one level further.

A bank row names its frame by a small ordinal and its star by an opaque number of SkyCruncher's own. It repeats neither the frame's facts nor the star's. Closed vocabularies such as status and reason are stored as one-byte codes, and the word list travels in the file's metadata, so a reader that meets a newer word still reads every column.

Versioning

• Every table and JSON product carries a major.minor.patch version. Adding a column or field is a minor bump, and an older file still reads with the new column absent, never filled with a zero.
• A change of meaning is a bump even when no column moves. A change of shape is a major bump: the bank went to 2.0.0 when its five fixed correction terms became one list column.
• The run receipt stamps the version of every table the run wrote, and names the catalog release the star identifiers came from.
• Version strings sit in the host digest, not the decision digest, so bumping a version does not make two identical measurements look different.

How it streams

• Frames and color planes can be written in Arrow's stream format, many record batches, each holding a run of whole image rows. A reader holds one batch at a time. The engine switches to this path when the data to hold exceeds half of the free memory it measured at startup.
• The bank is append-only. A deposit adds one new file per frame and never edits a row. Re-measuring a frame adds new rows stamped with the build that made them.
• Reading one star does not load the bank. Every 256 deposits the bank seals a curve segment, rewritten star by star with eight stars per record batch. An index says which batch holds a star, so one star's history costs one seek per segment.
• Reading one column of any table means reading that column's buffers only. That is what columnar storage buys.

Stars appear at two levels of detail. photometry.arrow is the full per-frame record a scientist or a debugger reads. The bank deposit is the lean archival row that light curves are built from.

The bank deposit row, every column

One row is one star, on one frame, in one band. This is the record of truth, bank schema 2.0.0.

Adding up the column widths gives about 143 bytes a row before file framing. The real M 31 deposit measured 146.6 bytes a row, so the difference is the file header, the schema metadata and padding.

Sky position is reached through the star: matches.arrow links each star_id to a Gaia source_id and the catalog release. The walk's confirmed.arrow also carries the catalog ra_deg and dec_deg, the predicted and measured pixel positions, and the residual in arcseconds.

The full per-frame row: photometry.arrow

skycruncher.photometry version 1.5.0 has 53 columns per star per band. They fall into these groups:

Status words here are measured, not on plane, uncataloged candidate and uncataloged rejected. Most numbers are 64-bit floats, so this table is wide: about 460 bytes a row on the Canon test frame. It is built for diagnosis, not for archiving.

These numbers do not all use the same frame, so compare within a row, not across rows.

Planned, and decided but not built

• No picture on the receipt. Decided 2026-09-11: the receipt carries data only. If the user saves or hosts a full image, the receipt gets a link to it. The link field is not in the run receipt yet.
• The residual map is regenerated, not stored. Decided 2026-09-11, along with landing the receipt on the user's own disk by default.
• Upload is opt-in. Nothing leaves the user's machine until they say yes once at first run. An export carries a scrub receipt (built, version 1.0.0) that says what was removed.
• Complete error bars in the bank. A bank row's bar carries only the flux fit's own sigma today. The brightness surface measures an excess scatter that is not in the bar yet.

Nothing here needs SkyCruncher's code. The tables are standard Arrow IPC files and the receipts are plain JSON. The column names below are real; the paths follow the run directory layout.

Engine tables keep their name, version and settings as one sorted JSON object under the metadata key skycruncher. Bank files carry the word lists that decode their one-byte codes (bands, status words, reasons, builds) in their own metadata, so a reader turns codes back into words without SkyCruncher's code.

A receipt is a measurement, not a copy. A frame is what is on the sky (stars and a galaxy) pushed through the camera's optics and the night's air, plus the glow of the air itself, plus noise. Each part can be held a different way: as rows, as a few numbers per frame, as a model kept once, or not at all. Pick a term to see it alone, what it costs, and which level holds it.

One frame, redrawn from its receipt

The left picture is a real 10-second sub of M 31. The right picture was drawn without any of its pixels, from the receipt the Layers tab proposes: every Gaia star at its catalog position with the brightness this frame measured, drawn through this frame's optics and air, on its sky glow, with noise at the predicted level. The bottom picture is the real frame minus the drawing (without its noise), so whatever is left there is what the receipt does not hold.

The three choices add a galaxy layer. Galaxy model is M 31, M 32 and M 110 as the night's other 351 subs saw them, with the stars removed: an object model held once, pushed through this frame's optics and air, and only its brightness is new in this frame (16 bytes). Survey projection takes the DSS2 survey plates of the same sky instead, projects them through our camera, and scales them with two numbers per color plane: the version every camera could share today, but the plates are saturated in M 31's bulge. Either way the galaxy is held once, in one shared place, and is not baked into the frame.

What a receipt keeps

If you keep only receipts, here is what you can and cannot get back.

Raw frames keep everything and cost the most. A receipt keeps the part of a frame that a star survey needs, in a layout built for asking questions across thousands of frames. The trade only works if the measurement is good, which is why every row carries its error bar, its evidence and the build that made it.

One more limit, stated in the code itself: the first community bank's error bars were 20 to 50 times smaller than the scatter the points actually showed, because only one term went into them. The bank now names which terms are in each frame's bars, so a reader can tell a small bar from an incomplete one. Part two depends on getting this right.

Each level stores what the level above it cannot predict. A frame becomes what the session model predicts, plus whatever surprised it. A session becomes what the hardware profile and the star database predict, plus whatever surprised them. Once the models are good, nearly every star on nearly every frame is unsurprising. Then you keep a few running totals per star instead of a row per measurement, and full detail only for the surprises: variables, transients, artifacts. Galaxies and nebulae follow the same rule through object models, which sit beside the star database (section 12).

For every star on every frame, the levels above make a prediction: a value and the noise it should have. If the observed value matches within that noise, nothing is stored for this frame. The star's running totals are updated instead. If it does not match, the measurement is stored in full as an exception.

Bright stars need their own path. A failed fit is not a surprise about the sky, so a bright star whose flux comes back missing, negative or far from its catalog magnitude should be predicted from the catalog and its saturation model, and the row flagged, rather than stored as an exception (section 06 shows why).

Per star, per band, per hardware unit, the design keeps three sufficient statistics: the count, the weighted sum of residuals, and the weighted sum of squared residuals. They rebuild the mean and the scatter exactly. They also add: session totals add into hardware totals, which add into the star's global totals. The rollup is just addition, and it can be redone in any order.

The same normalization as today, carried one level further. Facts live at the level they belong to, and everything below points up by key.

The noise model is everything. "Within the predicted noise" only means something if the error bars are right. Today's bank shows the danger: the first community bank's error bars came out 20 to 50 times too small. Too small, and everything looks surprising and nothing compresses. Too large, and real variability is averaged away.

• Prove the noise model first. A hardware profile must show its noise model holds on data it was not tuned on before the system stops recording detail for that unit.
• Keep an audit sample. Full rows for about 1 in 50 frames, forever. It catches model drift and lets a new model be tested against real rows.
• Version the models. Exceptions and totals only mean something against the model version that produced them, so each carries it.
• Fit hardware and stars together. The hardware is calibrated from constant stars, and star values come from calibrated hardware. Large surveys solve this by fitting both at once, and Gaia anchors most stars.
• Model the noise on native pixels. On M 31, formal errors came out 2 to 3 times too small because debayering and resampling correlate neighboring pixels. So all modeling stays on the raw Bayer planes (section 14).
• Admit new model terms only on held-out data. A term earns its place only if it predicts frames it was not fitted on, and outside catalogs may pull a value toward zero but never put light where none is measured. Section 14 lists both rules.
• Never drop a bright star. Carry each one as a flag, its catalog position and magnitude, and a model of its core, wings and saturation. On the M 31 sub, 47 of 400 band rows for stars brighter than G 10 came back with no usable flux, all of them in groups where an uncataloged detection overlapped the star (section 06).
• Know what is lost. The exact noise of each constant star on each frame. That is fine for science if the model is right, but it limits questions nobody has asked yet. That is why the audit sample and the raw-frame retention policy matter.

A receipt today keeps the stars and drops the light between them. The same predict-and-store rule can bring that light back. Model each galaxy or nebula once, as an object model. Project it into every frame through the same hardware and session profiles that predict the stars. Then store only what the projection could not predict.

Tried on a real frame. The figure in section 06 does this on one M 31 sub. An object model built once from the night's other 351 subs (481 kB held once, 16 bytes per frame) brings back 96% of the galaxy light and leaves structure 1.12 times blank-sky noise. A smooth bulge-and-disk fit in its place leaves 1.6 times (1.48 with GALFIT, 1.33 with M 31's bar). The DSS2 survey plates projected through the frame (47.9 MB held once, 8 numbers per frame) take out the disk but leave the bulge, where the plates are saturated (7.0 times). Storing the same galaxy as pixels would cost 2.29 MB in every frame. One shared reference, held in one central library, plus a few bytes per frame replaces the galaxy's pixels in every frame that sees it: the object model pays for itself in the first frame, and the survey plates after about 21 frames of M 31.

1. The object model. A map of the object in each band, fixed on the sky. It can be a survey image (Pan-STARRS, Hubble, IPHAS H-alpha), a layer fitted jointly from SkyCruncher's own subs, or a fitted profile (a bulge and a disk, or a Sérsic profile) with a map for what the profile misses. Bright stars inside it come from the star database, never from the map.
2. Geometry. The frame's solved pose places the model on the sensor, pixel by pixel.
3. Blur. The model is convolved with the frame's star shape (PSF): the hardware profile's shape across the field, widened by the session's seeing and smeared by the mount's field rotation during the sub.
4. Band and scale. The hardware profile's color response maps the model's band onto the camera's; the frame's zero point scales it.
5. Sky. The session's sky plane is added. The result is the predicted frame, before noise.

If exceptions run to a few percent of rows, the cost of a star on a frame drops from about 590 bytes to a few bytes of totals plus the occasional exception row. That is plausibly one to two more orders of magnitude beyond today's 7.5 to 1. The real figure depends on how often the model is surprised. The chart below compares four ways to keep a growing community's data over ten years. Both rollups include the extended-light layer of section 12 unless you switch it off below.

One direction goes further than receipts: archive frames as forward-model parameters plus a residual quantized at the noise floor. Named model components compete to explain each pixel, and each one is admitted only if it earns its place. Storage becomes minimum description length: the cost of the model's parameters plus the cost of whatever the model leaves unexplained. That is the same rule as Part two, applied to pixels instead of star measurements. These are the ideas that bear on this page.

Formats, schemas and byte counts are read from SkyCruncher's engine and its measurement reports as of October 2026; the code is not public yet. The M 31 frame in Part one is the author's own: one 10 s sub from a ZWO Seestar S30 Pro on the night of 2026-09-18, with the night's other 351 subs for the object model. The M 31 findings in section 12 also use data from the Seestar Collaboration's M 31 project, with permission.