A thin line beside a passage says how far its claim is proven: measured, published, prototyped, planned, illustrative, inferred (an estimate says so in words). Colored numbers carry the same mark. A passage with no line is stated. How claims are marked.
The short version
- One format for every kind of light. Nebula glow, dust, galaxies and the haze of stars too faint to see one by one all go on the same grid, so they can be compared cell by cell. (planned)
- Small, without throwing light away. Three narrowband maps of the northern sky take about 18 % of the data they replace, with the total light exact and faint filaments intact. (measured)
- Read only as deep as you need. Detail is stored coarse to fine, so a wide-field telescope can stop reading long before a sharper one does. (planned)
- Nights of frames become one more survey. Each night's results are written onto the same cells, as sums that add across nights, cameras and people. (planned)
- The first maps took hours, not months. One cloud machine, a single computer in a data center, stored all three maps and their noise maps in about 3.7 hours. (measured)
None of this is in the SkyCruncher app yet. The app is in development, and the maps are stored privately while I check each survey's terms, so nothing made from them is published here. What isn't done yet has the rest.
Why one shared map
Most image pipelines keep each stage in private coordinates: sensor pixels here, a flattened mosaic there, a catalog somewhere else. Every handoff means resampling, and every resampling adds small errors of its own. This design puts everything onto one grid that much of astronomy already uses. That also keeps a frame receipt (the record of what was measured on one frame) from becoming a private format: any survey can be pulled into it, and any other dataset compared with it.
Open standards, not a black box
HiPS (Hierarchical Progressive Surveys) and MOC (Multi-Order Coverage maps) are IVOA standards built on nested HEALPix. A layer or a coverage map exports to them with no resampling, and Aladin and other viewers read them natively.
Any survey joins on the same ids
A survey on nested HEALPix lines up cell for cell with ours. Comparing two datasets is a join on cell ids, not a reprojection with its own errors.
Catalogs speak it too
HATS (HEALPix Adaptive Tiling Scheme) partitions big catalogs, Gaia and Pan-STARRS among them, by HEALPix cell. HATS catalogs drop onto the same cells as our layers.
Synthetic skies sit next to the engine
Test skies are drawn from the same tree the engine reads, so they are faster to make and closer to what the engine sees.
Residuals in the same language
What is left after a fit is stored on the same cells, so a residual from tonight compares directly with any layer or survey.
Precision is just depth
Sub-pixel detail needs no special format. A finer position or a finer feature is one more level down the same tree, to order 29 if needed.
HEALPix in plain words
HEALPix (Hierarchical Equal Area isoLatitude Pixelization) cuts the whole sphere into 12 base cells of exactly equal area. Each cell splits into 4 children, and those split again, as many times as you like. Satellite maps of the cosmic microwave background use it, and so does Gaia. (published)
The number of splits is the order. Order 0 has 12 cells, and each order has four times as many as the one before. An order-15 cell is about 6.4 arcseconds across (an arcsecond is 1/3600 of a degree), close to one pixel of a Seestar S30 Pro, which covers 3.67 arcseconds. The full Moon is about 1,860 arcseconds wide.
Because every cell splits into exactly four, the numbering nests: the children of cell i are 4i to 4i + 3. Zooming in or out is arithmetic on the cell number, and a parent's value is exactly the mean of its four children, so no light is created or lost going up or down.
Show every order in a table
| Order | Cell side | Cells, all sky | Plain all-sky map | What sits here |
|---|---|---|---|---|
0 | 58.6° | 12 | 48 B | – |
3 | 7.33° | 768 | 3.1 kB | Storage region: packs are grouped per order-3 cell (768 regions) |
5 | 1.83° | 12,288 | 49.2 kB | – |
7 | 27′ | 196,608 | 786.4 kB | File unit: each order-7 cell holds its own subtree |
10 | 3.4′ | 12.6 million | 50.3 MB | Finkbeiner H-alpha native |
11 | 1.7′ | 50.3 million | 201.3 MB | Planck and CSFD dust maps native |
12 | 51.53″ | 201.3 million | 805.3 MB | – |
14 | 12.88″ | 3.2 billion | 12.9 GB | WSSA 12 µm dust native |
15 | 6.44″ | 12.9 billion | 51.5 GB | NSNS narrowband pixel (6.4″) |
16 | 3.22″ | 51.5 billion | 206.2 GB | S30 Pro pixel scale (3.67″) sits between 15 and 16 |
17 | 1.61″ | 206.2 billion | 824.6 GB | – |
18 | 805.2 mas | 824.6 billion | 3.30 TB | Seestar optics: the deepest order where structure is |
20 | 201.3 mas | 13.2 trillion | never stored dense | – |
24 | 12.6 mas | 3.4 quadrillion | never stored dense | – |
29 | 0.39 mas | 3.5 quintillion | never stored dense | Finest id: star positions as a cell plus offset |
Computed from the grid's definition. A plain map stores one 4-byte number per cell. At order 18 that is 3.3 TB for a single map of the sky, which is why this design never stores one.
The light map Note explains how a single frame reads a map like this. This Note is about how the map itself is stored and filled.
The tree: store differences, keep every level
A layer is one map of one kind of light: hydrogen-alpha glow, say, or dust. Each layer is stored as a tree. A cell stores its mean once. Each finer level stores only how the four children differ from their parent, which takes three numbers, because the fourth follows from the others. A reader walks down from the top, adding differences, until it reaches the depth it needs. Empty sky stays cheap, because its differences are nearly all zero.
The obvious way to save more space is to drop any difference smaller than its noise. I tested that and it loses faint filaments. A thin streak can fail every single-cell test and still stand out to a search along its length. On synthetic filaments, a store that dropped those differences kept only 0 to 14 % of each filament's cells. (measured)
So every level is kept, as small whole numbers rounded to a fraction of the noise, and runs of zeros compress to almost nothing. Near galaxies, nebulae and dark clouds the rounding is finer, and it eases off smoothly with distance. Inside those protected areas, the rounded map found exactly what the untouched map found in all 24 test cases; in open sky it differed in 3 of 24. (measured)
Noise has to be measured at each level, too. At coarse levels it ran 9 to 22 times what simple white noise would predict, so a single noise figure would have spent detail on noise. (measured)
What the layers hold
The stack holds hydrogen-alpha and [O III] (light from specific gases in nebulae), broadband continuum, dust, galaxies and unresolved light. Stars never go in a tree: they stay as catalog points at exact positions. Each source is its own release with its own license and credit, so terms never mix by accident. (planned)
Surveys and my own frames can be merged level by level. Wide, calibrated surveys are strongest at coarse levels and my frames at fine ones, so each level is weighted by its measured noise and the crossover falls wherever the noise curves cross. Nobody picks it by hand. (planned)
The stars you can't see one by one
Below every image's detection limit sits a haze of stars and galaxies too faint to pick out. It is easy to mistake for sky glow, dust or nebulosity. The plan is to store it as one more layer, rolled up from catalogs. (planned)
"Too faint" depends on who is looking. A star lost in one short exposure shows up in a deep stack. So each cell stores the predicted light of catalog sources per magnitude bin (magnitude is the astronomer's brightness scale, where bigger means fainter). A reader sums the bins fainter than its own limit, so a single frame and a deep stack each get their own unresolved sky from one layer. (planned)
The surveys, with their terms
Every source below belongs to its team. "To confirm" means I have a second-hand reading of the terms and will check the source's own statement before any release. "Stored" means a layer sits in my private storage. Nothing here is published or offered for download yet.
| Source | What | Sharpness | Order | License and credit | Status |
|---|---|---|---|---|---|
NSNS DR0.2 | Hα, [O III], [S II], and a noise map for each | 6.4″ pixel | 15 (noise maps 14) | CC BY-NC-SA 4.0; credit the NSNS author | Storedwhole footprint stored, 2026-10-30 |
Finkbeiner 2003 | Hα composite, all sky | 6′ beam | 10 | Open, with citation | Storedfull sky stored |
WHAM-SS | Hα, velocity resolved | 1° beam | 7 | Citation requested | Storedfull sky stored |
SHASSA, VTSS | Hα wide-field | 0.8′–1.6′ | 11–12 | Citation requested | Planned |
MDW | Hα, northern sky | ≈ 6″ | 15–16 | CC BY on the catalog; no license found for the images, asking the team | First pass, in the viewerreprocessed by me, not an MDW product, rough on faint sky: known issues |
Planck GNILC, Planck free-free, CSFD, SFD | Dust and free-free | 5′–6′ | 11 | Planck Legacy Archive terms; CSFD CC BY 4.0 | StoredGNILC dust emission and optical depth, free-free and CSFD stored full sky; SFD not yet |
WSSA 12 µm | Diffuse dust, IFN tracer | 15″ | 14 | Citation (to confirm) | Not in the first setout of the core set: its only route is CDS, which is not answering |
Edenhofer 2024, Bayestar19 | 3D dust cubes | ≈ 3′–14′ | 8–10 | CC BY 4.0; CC0 | Planned |
IPHAS DR2, VPHAS+ | Galactic-plane Hα, broadband | ≈ 1″ seeing | 18 | VPHAS+ ESO CC BY 4.0; IPHAS to confirm | Plannedthe survey teams' own servers answer from the cloud; a later run |
SGA | Galaxy outlines and profiles | per galaxy | native | Open (to confirm) | Storedused in the mask |
Legacy Surveys | g, r, z (+ Tractor models) | 0.26″ pixel | 15, deeper in galaxies | CC BY 4.0 (to confirm) | Plannedafter star removal |
Pan-STARRS DR2 | g, r, i, z, y (+ catalogs) | 0.25″ pixel | 15 everywhere; 18 inside galaxies at least 2′ across | STScI/MAST terms (to confirm) | Plannedscope set, processing waits; star removal tested on two patches |
Gaia DR3 | Stars, XP spectra | positions at epoch 2016.0 | point catalog | ESA/Gaia/DPAC acknowledgement | Designed |
NSNS is the Northern Sky Narrowband Survey, the only wide-area source of [O III] and [S II] light; it covers about 65 % of the sky. Its terms are non-commercial and share-alike, so anything made from it carries those terms.
How long it takes
Cloud work here is sized in time on one machine.
| Job | How long | On what | Status | Notes |
|---|---|---|---|---|
The three narrowband lines and their noise maps (NSNS, its whole footprint) | about 3.7 hours, start to finish | One cloud machine | Measured | 126 GB downloaded, 22.5 GB stored. All 3,168 work units finished and verified; none failed. Estimated beforehand: 3.5 to 4 hours. |
A coarse all-sky map (dust, H-alpha) | 7 to 14 minutes a map | One cloud machine, 8 processor cores | Measured | Timed on four maps in a test run. The run that stored the final six wasn't timed map by map. |
Galactic-plane surveys (IPHAS, VPHAS+) | No estimate yet | Not run | Planned | Their servers answer; sizes and times aren't known until a test run. |
Broadband surveys (Legacy Surveys, Pan-STARRS) | No estimate yet | Not run | Planned | Waiting on star removal at scale. Pan-STARRS is parked while the rest of the system is built. |
Downloads set the pace, not computing. In a speed test the machine pulled 138 to 360 MB/s from a large provider, while survey servers gave at most about 16 MB/s. I keep to a few connections per server, because many of these archives are run by small teams. (measured)
Encoding runs alongside the downloads. For the narrowband job I estimated about 31 billion cells and about 11 processor-core hours beforehand, a bit under 2 hours on 8 cores. (inferred)
What has been measured
- Exact round trip. Decoding gives back, bit for bit, what was encoded. (measured)
- Light conserved. Over the whole narrowband run, the largest relative error in a cell's total light was about 6 parts in 1016, which is rounding error in the computer's arithmetic. (measured)
- No seams. The run left no holes where storage regions meet, and an earlier test found differences across region edges matching those inside a region. (measured)
- Size. 17.4 to 18.0 % of the data downloaded for the three lines: H-alpha 5.97 GB, [O III] 5.85 GB and [S II] 6.04 GB. (measured)
Where my own frames come in
The goal is for each night's results to be written onto the same cells, at order 18 for Seestar S30 and S50 optics, only where the night shows real structure. Sums add across nights, cameras and people, and subtract exactly if a night is withdrawn. A result also names the layer versions it read, so a later, better layer never changes an old result. (planned)
The fit itself doesn't run on HEALPix cells. I tested it: solving directly on order-18 cells took 4.7 times the grid points, 1.6 times the iterations and 2.3 times the graphics memory of a grid aligned to the camera's sensor. So the fit runs on the sensor's grid and the results are filed on HEALPix. (measured)
Frame receipts filed by cell could shrink to an estimated 4 to 12 KB a frame, from roughly 0.3 to 0.5 MB today. That waits on the new fit. (inferred)
What this could make possible
These are possibilities, not results.
Light curves as a lookup
A 30 to 50 mm telescope collects hundreds of times less light than a professional 1 m one. What many of them could offer instead is steady coverage from many longitudes, minute by minute, of brighter variable stars. With every frame on the same cells, a light curve becomes a lookup.
Keeping star positions current
Gaia's published positions are for the start of 2016, and roughly a fifth of its sources have no measured motion. Repeated, current positions from many small telescopes could add something there. They can't beat Gaia on bright stars or on its reference frame.
What isn't done yet
- The app. None of this is in it yet, and it has no release date. (planned)
- The measured values aren't published. The sky viewer shows display images made from these maps, with each layer's terms and what's still open on them. The maps themselves stay private. (planned)
- Broadband layers. Images full of stars compress far worse, so stars have to come out first. The removal recipe has been tried on two test patches only, a nebular field and a crowded Milky Way field. (planned)
- Pan-STARRS is not in use. The plan is order 15 everywhere and order 18 inside large galaxies, never the whole sky at full depth. It is parked for now. (planned)
- One survey source has no working route. WSSA's only copy sits on a server that isn't answering. (planned)
- The fit on these cells. Writing nightly results to order 18 waits on the fit producing them, and it still needs a fast method for frames that rotate through the night. (planned)
- Known wobbles. Small sources at the edge of a survey's coverage came back slightly bright in two of three tests, and rounding can jump where one noise estimate meets the next. Both are being looked at.
- Survey dependence. NSNS rests on a single server, so I want a second copy worth keeping.
What changed
- This Note went up. The same day, the three narrowband maps and their noise maps were stored over NSNS's whole footprint in about 3.7 hours. The first set of surveys now excludes MDW and WSSA, and tests settled that the nightly fit stays on the sensor's grid. Later that day, a first pass of MDW was stored after all and went into the sky viewer with its known issues.
- First measurements of the tree, on a test area around the North America Nebula: exact round trips, and faint filaments surviving the rounding.
Read more
- The light map: how a frame reads a map like this.
- Frame receipt format: what gets recorded about each frame.
- All Learn Notes
- Standards: HEALPix, HiPS, MOC, HATS. Viewer: Aladin.
HEALPix: K. M. Górski et al., 2005, The Astrophysical Journal 622, 759. Survey credits and licenses are in the table above; each survey's data belongs to its team.