CMSSW side: what ember runs¶
Ember reads NanoAOD. It gets it in one of two ways:
- NanoAOD that CMS already makes is read as it is, with no CMSSW job: standard NanoAOD (any
NANOflavour, for example from MiniAOD), and the HGCal tables ofNANO:@HGCAL(DPGAnalysis/HGCalNanoAOD). - For a reconstruction file (RECO, FEVT, AOD)
emberwrites a cmsRun configuration of its own,ember_cfg.py, from modules that your CMSSW release already has, runs it, and converts its output. A MiniAOD gets standard NanoAOD instead (cmsDriver.py -s NANO).
Nothing has to be merged in CMSSW for this to work. The modules are CMSSW's own NanoAOD table producers, so the
output is plain NanoAOD, which you can read with uproot or ROOT like any other. Two CMSSW branches, not merged yet,
add generic table producers for what the release cannot write: the hits of each cluster, and the truth graph
(below). ember uses them when your release has them.
The code is embertools/cmsrun.py (the configuration) and embertools/export.py (the settings and the job). This
page describes them as tested with the integration build CMSSW_20_1_X_2026-11-17-2300.
How ember writes the configuration¶
What goes in ember_cfg.py is decided from three things:
- what your release has: its module plugins, from
edmPluginDump -a(0.1 s, so it is not cached); - what the input holds: its products, from the branch names of its
Eventstree; - the era: Phase 2 (HGCal, TICL, the ECAL barrel, HCAL) or Run 3 (also the ECAL endcap, the preshower and the
HCAL endcap). The era, geometry and global tag are found from the file, or given with
--era,--geometryand--conditions(How-to).
A table whose producer the release lacks, or whose collection the input lacks, is left out. The header of the
configuration lists each one, with the reason, and ember prints the same lines (left out: ...). Products of the
input whose class the release has no dictionary for (the SoA collections of a development release, for example) are
dropped on input, so that cmsRun can open the file. What was made from them stays.
--config FILE.py keeps the configuration. It is one readable Python file, with the input files, the number of
events and the settings written in it. cmsRun FILE.py runs it again in the same release, and
ember convert ember_nano.root -o NAME.ember converts its output. A failed job keeps it next to its log.
# The NanoAOD that Ember reads, from a CMSSW reconstruction file, with modules of this release only.
# Written by embertools 0.2.0; keep it, change it, and run it again with: cmsRun ember_cfg.py
# Then: ember convert ember_nano.root -o NAME.ember
#
# Phase 2 simulation: era Phase2C22I13M9, geometry ExtendedRun4D104, conditions 201X_mcRun4_realistic_v1
# Left out (not in the input, or not in this release):
# - the hits of clusters: this release has no CaloClusterHitsTableProducer, PFClusterHitsTableProducer,
# SuperClusterHitsTableProducer (add them with git cms-merge-topic mmarriot:nanoaod-cluster-hits)
# - this release has no TruthGraphFlatTableProducer, TruthGraphHitFlatTableProducer (add them with git
# cms-merge-topic mmarriot:truthgraph-nano-tables)
import FWCore.ParameterSet.Config as cms
...
What it writes¶
| part | tables | module (package) | era |
|---|---|---|---|
| HGCal hits | rechits: every rechit of CE-E and CE-H (silicon and scintillator) with its DetId, energy, time and position (its rechit_radius is left out by the profile: it is not a cell size) |
RecHitFlatTableProducer (PhysicsTools/TruthInfo), positions from TICLGeom (RecoTICL/Geometry) |
Phase 2 |
| ECAL and HCAL hits | pfrechits: the rechits above the particle-flow thresholds, with DetId, energy, time and position. The converter splits it into EBHits, EEHits, ESHits, HBHits, HEHits, HOHits and HFHits. |
PFRecHitFlatTableProducer (PhysicsTools/TruthInfo), positions from CaloGeometry. When the input does not keep the PF rechits (Run 3 RECO keeps only those that cleaning removed), the reconstruction's own producers make them again in the job. |
both |
| HGCal reconstruction | HGCalLayerClusters; one table per trackster collection of the input, named after it (for example ticlTrackstersCLUE3DHigh, ticlTracksterLinks, ticlCandidate), each with its list of layer clusters; TICLCandidates, with their tracksters and tracks |
the tables that NANO:@HGCAL writes (DPGAnalysis/HGCalNanoAOD), with only the columns that are drawn |
Phase 2 |
| PF clusters | PFClusterECAL, PFClusterHCAL, PFClusterHF, PFClusterHO, and PFClusterPS in Run 3 |
LayerClustersCollectionTableProducer (DPGAnalysis/HGCalNanoAOD), a table of any reco::CaloCluster collection |
both |
| Superclusters | SuperClusterEB; SuperClusterEE (Run 3); SuperClusterHGCal (Phase 2) |
SimpleSuperclusterFlatTableProducer (PhysicsTools/NanoAOD) |
both |
| The hits of each cluster | for each cluster table X above and HGCalLayerClusters: a count nXHits on X, and a table XHits of detId and fraction, one row per (cluster, hit) |
CaloClusterHitsTableProducer, PFClusterHitsTableProducer, SuperClusterHitsTableProducer: the branch nanoaod-cluster-hits |
both |
| Tracks | GeneralTrack: the general tracks, with their momentum, reference point and outermost hit (and, in Phase 2 with TICL candidates, their HGCal entry point; with MTD, their time at the beam line t0, from tofPID); GsfTrack: the electrons' GSF tracks |
HGCalNanoAOD's track table (hgcalGeneralTracks_cfi); SimpleGsfTrackFlatTableProducer |
both |
| Vertices | PV: every primary vertex (row 0 the leading one), with its time in Phase 2; SV: the inclusive secondary vertices |
SimpleVertexFlatTableProducer |
both |
| PF candidates | PFCandidates (particleFlow): momentum, type, vertex, ECAL entrance, energies, time, PUPPI weight (puppi), and the rows of their track, GSF track, muon, electron and photon (trackIdx, gsfTrackIdx, muonIdx, electronIdx, photonIdx) |
SimplePFCandidateFlatTableProducer; the rows are the keys of the candidate's references (trackRef().key()), which point into the collections of those tables |
both |
| Muon system | DTSegment, CSCSegment, GEMSegment (global position and direction), GEMRecHit, RPCRecHit (global position), each with its DetId and its fields |
DPGAnalysis/MuonTools's table producers, with the global position configured | both |
| Physics objects | Jet (AK4 Puppi jets, or CHS jets where the file has no Puppi jets: uncorrected), JetCHS (the CHS jets, next to the Puppi ones), Tau (HPS: kinematics, decay mode, the row of the leading charged hadron in PFCandidates), PuppiMET, PFMET, Electron, Photon, Muon, ElectronHGC and PhotonHGC (the HGCal electrons and photons of Phase 2), OfflineBeamSpot. Uncut, so that index columns address their rows. Electrons and photons have the rows of their supercluster (superClusterEBIdx, superClusterEEIdx: the PF supercluster, parentSuperCluster(); superClusterIdx for HGCal), electrons of their track and GSF track, muons of their inner track. |
PhysicsTools/NanoAOD's Simple*FlatTableProducers on the RECO collections, with the names and columns that standard NanoAOD uses; the index columns from the references' keys (lazyEval, as the producer is that of reco::Candidate). With --pat: standard NanoAOD (below), without these index columns. |
both |
| Truth | the truth graph tables (below) | the branch truthgraph-nano-tables |
Phase 2 |
- The tables keep the order of the CMSSW collection they come from.
- The cluster-hit tables give each hit by its DetId. The converter finds the hit with that DetId in a hit table:
the profiles have one relation per hit table, all built from the same cluster-hit table (for example the ECAL PF
clusters' hits in
EBHitsand inEEHits). A hit below the PF thresholds is not inpfrechits, and its link is dropped. - No table carries a cell size. The converter works it out from the DetId (Profiles).
- The float columns that
emberconfigures keep only the mantissa bits they need, 6 to 14 of the 23, chosen per column: 14 for positions (about 0.3 mm at 5 m), 12 for cluster energies, η and φ, 10 for pT, times, axes and widths, 8 for fractions, 6 for the jets' energy fractions. The TruthInfo hit tables have no precision option, and keep all 23. - The NanoAOD file is temporary, so it is written with LZ4, which is quick to write and to read back.
Physics objects, and PAT¶
RECO has no jet energy corrections, jet IDs or b-tag tables that a NanoAOD table producer of the release can read.
So, by default, the physics objects come straight from the RECO collections: uncorrected jets with no IDs (b-tags with
the branch nanoaod-jet-tags), and HPS taus with their decay mode.
These tables cost under 1 ms per event.
--pat takes the physics objects from PAT and standard NanoAOD instead. ember writes the configuration of
cmsDriver.py ember -s PAT,NANO --processName EMBER, and appends its own tables of the hits, clusters, HGCal objects
and tracks to it, in the same output file. The bundle then has:
- corrected jets with their IDs and b-tags, and taus (
Tau); PV,SVand MET with the columns of standard NanoAOD;- one
Electronand onePhotontable, as standard NanoAOD has: noElectronHGCorPhotonHGC.
It costs (Phase-2 ttbar, 10 events, 4 threads):
ember's tables, whole job |
PAT and NANO alone | |
|---|---|---|
| time per event, no pile-up | 0.57 s | 0.85 s |
| time per event, 200 pile-up | 0.71 s | 2.25 s |
| time to start | 19 s | about 32 s |
| memory, 200 pile-up | 3.3 GB | 8.6 GB |
End to end with --pat, 5 events took 55 s and 5.5 GB on Phase-2 ttbar without pile-up, and 52 s and 3.5 GB on Run 3
RECO.
A MiniAOD always gets standard NanoAOD (cmsDriver.py -s NANO): it has no hits or clusters to add. With a MiniAOD or
--pat, when the file has a trigger that was run again (reHLT, in RelVals), the trigger objects are read from it.
Truth¶
Truth in Ember is CMSSW's truth graph (truth::Graph, PhysicsTools/TruthInfo), its hit index, and the reco ↔ truth
associations of SimGeneral/TruthGraphAssociatorProducers, and nothing else. CaloParticles, SimClusters and
TrackingParticles are not used. Truth graph explains why, and what the graph still lacks.
The tables. The release has no NanoAOD tables of the truth graph. The branch truthgraph-nano-tables adds them.
When your release has them and the input holds the graph, ember_cfg.py writes:
TruthParticles,TruthVertices,TruthParticleChildren: the signal interaction, the particles that reco objects are matched to, every particle that entered the calorimeter (so that each hit's simulated energy has its particle), and the ancestors of all of them; their production and end vertices; the parent → child links;TruthHGCalRecHitsandTruthPFRecHits: the simulated energy that each of these particles and its descendants left in each hit ofrechitsandpfrechits, from the graph's hit index;<Key>ToTruthandTruthTo<Key>, with their...Linkstables: the reco ↔ truth associations in both directions, for the collections that the release's label registry names: general tracks, the trackster collections, ECAL and HCAL PF clusters, and the primary and secondary vertices (OfflinePrimaryVerticesToTruth, ..., with the truth vertices they match kept). The vertex associations are written when the associators run in the job, not from an input's own maps.
The associations come from the maps the input holds, when it has them (the TICL v6 samples were made with the
associators), and no associator runs; the header names the tables whose maps the input lacks. Otherwise CMSSW's
associators run in the job. --no-truth-associators skips
them: the bundle then has the graph and the hit energies, but no reco ↔ truth matches.
Which inputs have it. The Phase-2 eras carry the process modifier enableTruth. With it, the DIGI step builds the
graph and its hit index, and FEVTDEBUGHLT and RECOSIM keep them. So a stock Phase-2 step3 file has them, with the
pile-up of the in-time bunch crossing only (the TICL v6 samples have all crossings; the profile keeps the in-time
one). AOD, MiniAOD, NanoAOD, premixed pile-up, FastSim and Run 3 files have no truth graph. For Phase-2 simulation
without it, the header of ember_cfg.py says so.
What it costs, on Phase-2 ttbar at 200 pile-up (3 events, 4 threads): the job takes 3.1 s per event with the
associators, 2.1 s without them (--no-truth-associators), and 0.7 s without truth. Summed over the threads, the
associators take 1.8 s per event for the tracks, 1.0 s for the tracksters and 0.25 s for the PF clusters (3 s of CPU
for 1 s of wall time, as they run beside the rest), and the truth tables 0.14 s; the rest is reading the stored graph
and hit index. The bundle is 17.0 MB per event instead of 5.9 (both with the cluster hits), and 14.0 MB without the
associators.
Without pile-up, a job of 10 events takes 29 s with truth and 28 s without.
The CMSSW branches¶
All are generic NanoAOD table producers, useful for truth-graph studies, validation and ML training without Ember.
Add them to a CMSSW_20_1_X area with git cms-merge-topic (How-to).
| branch | what it adds | state |
|---|---|---|
mmarriot:nanoaod-cluster-hits |
PhysicsTools/NanoAOD: CaloClusterHitsTableProducer, PFClusterHitsTableProducer, SuperClusterHitsTableProducer. For any reco::CaloCluster, reco::PFCluster or reco::SuperCluster collection, a hit count on the cluster table, and a table of detId and fraction (from hitsAndFractions()). At 200 pile-up, the layer-cluster hits take 0.006 s per event. |
pushed; the pull request to CMSSW is not open yet |
mmarriot:truthgraph-nano-tables |
PhysicsTools/TruthInfo and SimGeneral/TruthGraphAssociatorProducers: the truth graph tables above (TruthGraphFlatTableProducer, TruthGraphHitFlatTableProducer, truthGraphNanoTables_cff) |
pushed; it builds on cms-sw/cmssw#52451 and contains its commits. Its pull request follows when #52451 is merged. |
mmarriot:nanoaod-calo-rechits |
every ECAL (EB, EE, ES) and HCAL (HBHE, HF, HO) rechit with the position of its cell: RecHitEB, ..., RecHitHO (detId energy time x y z; EE and ES not in Phase 2, where those collections are empty). They replace the PF rechits, and the cluster hits are found in them. The names are HcalNanoAOD's: ember_cfg.py does not also write HcalNanoAOD's tables, which would collide. |
pushed |
mmarriot:nanoaod-track-hits |
the valid hits of any reco::Track collection of a RECO file (an AOD keeps none: left out, with a note): n<T>Hits on the track table T, and <T>Hits (detId x y z layer), for GeneralTrack, GsfTrack, and the standalone and global muon tracks (StandaloneMuonTrack, GlobalMuonTrack, written with their hits) |
pushed |
mmarriot:nanoaod-vertex-tracks |
PVTracks, SVTracks: (vertex, track) pairs with the track's weight |
pushed |
mmarriot:nanoaod-pfcandidate-links |
on PFCandidates: the rows of its track, GSF track, muon (checked against the collection each reference points into) and TICL candidate; PFCandidateClusters<ECAL, HCAL, HF, HO, PS>: (candidate, cluster) pairs from the PF blocks |
pushed |
mmarriot:nanoaod-reco-constituents |
JetPFCands, JetCHSPFCands (jetIdx, pfCandIdx), TauPFCands (tauIdx, pfCandIdx) |
pushed |
mmarriot:nanoaod-jet-tags |
the b-tags of the CHS jets as columns of JetCHS (btagDeepCSVB, btagJP; in RECO only the CHS jets have tags) |
pushed |
mmarriot:nanoaod-muon-track-links |
on Muon: the row of its standalone track (standaloneTrackIdx), checked against the collection the reference points into (standAloneMuons:UpdatedAtVtx, or standAloneMuons when the update at the vertex failed: then -1) |
pushed |
mmarriot:nanoaod-mtd-rechits |
RecHitBTL, RecHitETL: every MTD rechit with its time, time error, energy (MeV) and position, and its cell (row, column: an ETL DetId is the sensor's, so an ETL hit is its DetId with its cell) |
pushed |
mmarriot:muontools-rechits-chambers |
DPGAnalysis/MuonTools: DTRecHit, CSCRecHit with global positions; MuonChamber, the chambers that hold a rechit or a segment (centre, axes, half sizes), which the rechits and segments find by DetId; Muon<DT, CSC, GEM>Segments, each muon's segments |
pushed |
mmarriot:nanoaod-reco-tables-all |
all of the above together | pushed |
For each branch whose producers the release lacks, the configuration's header names the branch and the branch of all
of them. The table and column names above are those of the branches as pushed (reviewed, with their review fixes), and
ember reads them as they are. The pull request text of each branch is kept with the branches.
What it costs¶
Measured with the integration build CMSSW_20_1_X_2026-11-17-2300, 4 threads, 10 events, without truth. Phase 2 is
ttbar with geometry ExtendedRun4D104 and the stock reconstruction; Run 3 is JetMET 2025C RECO data, read from EOS.
| per event | Phase 2, no pile-up | Phase 2, 200 pile-up | Run 3 data |
|---|---|---|---|
| cmsRun: time to start (once per job) | 19 s | 20 s | 16 s |
| cmsRun: time per event | 0.57 s | 0.71 s | 0.24 s |
| NanoAOD | 0.69 MB | 7.7 MB | 0.79 MB |
| bundle | 0.40 MB | 5.4 MB | 0.21 MB |
| with the cluster hits and truth: bundle | 0.70 MB | 17.9 MB | (no truth) |
The PF candidates, CHS jets, taus, GSF tracks, every vertex and the muon system's tables (measured against the main
branch on the same events: 3 events at 200 pile-up, 10 otherwise) add 0.02 MB per event to the bundle without
pile-up, 0.46 MB at 200 pile-up (10k PF candidates) and 0.07 MB in Run 3. At 200 pile-up their modules take about
50 ms per event of the job's 0.7 s (TimeReport over 3 events: the PF candidate table 21 ms, which includes reading
particleFlow from the file, as the first module that asks for a product pays for its decompression; the HGCal
electrons 10 ms, whose index columns are evaluated by name for each object, lazyEval). Warm, a Run 3 job of 10
events takes 28 s against 27 s.
With every branch (nanoaod-reco-tables-all, cluster hits and truth included), on the stock ttbar files: the
bundle is 0.77 MB per event without pile-up and 19.7 MB at 200 pile-up (the RECO file itself: 8.0 and 177 MB). Their
largest additions at 200 pile-up, in the NanoAOD: the hits of the general tracks (1.15 MB), the MTD rechits (0.58 MB)
and the PF candidates (0.34 MB). Measured side by side on the same 3 events at 200 pile-up, 4 threads, on a loaded
machine: 5.4 s per event with every branch against 5.0 s with the cluster hits and truth only (8.7 → 11.0 s of
CPU, mostly reading more of the RECO file: track extras and hits, PF blocks, MTD hits). Most of either is the truth
associators (tracks 1.8 s, tracksters 1.0 s).
- The job start is most of the time for 10 events: loading the libraries, building the geometry and the conditions.
Before it,
embertakes a few seconds to find the settings and to check the input's classes. - A RECO event at 200 pile-up is 177 MB on disk, but the job takes 0.7 s for it. The source reads a product from the file only when a module asks for it, so most of the event (truth, trigger, digis) is never read or decompressed.
- At 200 pile-up with the cluster hits, the largest tables are the HGCal rechits (4.7 MB per event, at full precision), the hits of the layer clusters (1.6 MB), the layer clusters (1.1 MB) and the tracks (0.44 MB).
To see where the time goes in a job, add process.options.wantSummary = True to the configuration (a TimeReport per
module at the end of the log), and run it with cmsRun.
SoA collections¶
Offline reconstruction (step3, FEVTDEBUGHLT) of a stock release stores no SoA collections. SoA reaches a file only
from HLT or NGT outputs, or from samples made with extra keep rules (such as the TICL v6 development samples). The
release cannot write SoA collections as NanoAOD tables yet: that needs a generic SoA table producer, which waits for
cms-sw/cmssw#52491. So ember_cfg.py writes the legacy collections, and
lists the input's SoA collections as left out. Where a NanoAOD file has both, the converter writes one source per kind
of object (Profiles).
Planned: every hit and every object¶
The goal is that the viewer shows all the hits and all the objects of an event. This is what ember_cfg.py does not
write yet with the release alone, and how each one is planned. Bundles made with Ember's earlier export (for example the ttbar data sets on
https://marriot-evd.web.cern.ch) have most of them, through tables that are no longer made.
| not written yet | planned route |
|---|---|
| what the branches above add | the branches, as cms-sw pull requests |
| SoA tables | the generic SoA table producer (above) |
| exact HGCal cell outlines | see below |
| HGCal fine scintillator tiles (the v19 geometries: D120, D122, D123, D126–D128) | sized as 432 tiles around the ring from the DetId; still to check against every cell of such a geometry |
HGCal cells as drawn today. Each HGCal cell is drawn as a regular hexagon centred on the cell's position, with its corners at the cell's half-width (the apothem for silicon, r·Δφ/2 for a scintillator tile). So:
- every cell has the same orientation. The real cell lattice is rotated by 30° from it in most layers, and the rotation changes from layer to layer;
- a cell is drawn slightly smaller than it is (its corners where its sides are), so neighbours never overlap;
- partial cells at the edge of a wafer are drawn whole, and scintillator tiles are drawn as hexagons.
Positions, energies and the hits that each object owns are exact; only the outline of each cell is approximate. The plan: the converter takes each layer's lattice angle from the cell centres (rotation only, nothing in CMSSW), and exact outlines come later with a CMSSW producer of the cells' corners.
Requests for what is still missing are welcome in the issue tracker.