Hosting the site¶
Planned, not available. The CMS site and its service that this page describes do not exist yet: no EOS project space, no user areas, no shared samples kept by cron. Until they do,
ember FILE.rootwrites the bundle on your machine, and you open it at https://marriot-evd.web.cern.ch, a personal copy of the viewer with a few data sets (Getting started). The commands and scripts below exist, and were run against a local folder and a test folder on EOS. Everything marked unverified comes from the documentation of the CERN services, and has not been tried.
This page is for the people who will set up and run an Ember site. Users do not need it.
The site is a static website: a folder of files, served over HTTPS behind the CERN single sign-on. There is no
server process and no database. Commands write everything in it: each user's ember FILE.root, which publishes to
the site when it is configured (EMBER_SITE_DIR or --site), and two cron jobs.
This page gives the overview, and what to ask CERN IT for. The operating details (configuration keys, what each pass
does, how to redeploy) are in the repository's
service/README.md.
/eos/project/c/cms-ember/ and https://cms-ember.web.cern.ch are placeholders. They are set in one file
(embertools/site.py), and EMBER_SITE_DIR, EMBER_SITE_URL and EMBER_SERVICE_DIR override them.
Layout¶
/eos/project/c/cms-ember/
├── www/ the site (EMBER_SITE_DIR), served as https://cms-ember.web.cern.ch (EMBER_SITE_URL)
│ ├── index.html, assets/ the viewer, built by service/deploy.sh
│ ├── index.json the shared data sets, rewritten by `ember sync` (newest data first)
│ ├── whoami answers {"login": "..."} for the signed-in person (optional, see below)
│ └── data/
│ ├── <campaign>/ shared groups: bundles and campaign.json (titles), written by `ember sync` / `ember publish`
│ └── u/<login>/ each user's bundles and index.json, written by that user's `ember`
├── bin/ember the command as one file, for users (EMBER_BIN_DIR)
└── service/ EMBER_SERVICE_DIR
├── code/ what cron runs: embertools/, service/, pylib/, site.env
├── state/ lock/ while a sync runs, failures.json
├── logs/ sync.log, clean.log
└── cmssw/<release>/ CMSSW areas for the RelVal campaigns
A user's index has this form:
{"updated", "bundles": [{"file", "title", "events", "size", "created", "published", "expires"}]}
- The bundles are listed most recently published first.
fileis relative to the site, so a link is<EMBER_SITE_URL>/#b=<file>.createdis when the bundle was converted,publishedwhen it was last put on the site, andexpiresis 30 days afterpublished.
The site's own index.json never lists anything in data/u/. A user's bundle is seen only by its owner and by
whoever has its link.
What to ask for¶
A checklist for the request to CERN IT and CMS. All of it is unverified until it is done.
- An EOS project space (
/eos/project/c/cms-ember), through the "Request project space on EOS/CERNBox" form (CERNBox documentation). It comes with awwwfolder, and admin, writer and reader groups. For its size, see Quota. - A service account to own the site, the cron jobs and the files they write, so that the site does not depend on one person's account.
- A webEOS site (
cms-ember, or the name CMS prefers) that serveswww/, created in the Web Services Portal (webEOS documentation). By default a site is owned by a primary account. A site owned by a service account needs a request to lift this. - Access for CMS members only. Keep the site's Guest access off (the default: every visitor signs on), and
require a role. In the Application Portal, give the site's application a role mapped to the CMS members group.
Then either protect the whole site with that role, or enable Use .htaccess files and put this in
www/:(webEOS: authorization).AuthType openid-connect Require claim cern_roles:<role identifier> - Write access for CMS members under
data/u/(next section). - acron for the service account on lxplus (acrontab documentation).
- Optionally, CGI execution on the site, for
whoami.
Users' areas and their access¶
The first time a user publishes to the site (ember FILE.root with the site configured), ember creates
data/u/<login>/. It writes the bundle and the index there,
either through the mounted file system (lxplus) or through XRootD with the user's Kerberos ticket.
Set up these rights once on the site folder, with EOS ACLs:
- everyone who may see the site can read (the web server's account included);
- a CMS member can create their own folder under
data/u/and write in it, and in no one else's; - the service account that runs the cron jobs can write everywhere. It removes expired bundles and rewrites the lists.
These rights are the only thing that keeps one user from replacing or removing another user's bundles. ember
writes as whoever runs it, with their own EOS rights. The site has no server that could check anything. --user and
EMBER_USER name an area; they prove nothing.
What ember does check:
- a login and a bundle name are plain names, so that a path built from them stays inside
data/u/<login>/. A login starts with a letter, followed by letters, digits,-and_, 2 to 32 characters in all, in lower case; ember cleanleaves alone anything indata/u/that is a link.
Unverified: nobody has yet worked out or tried the ACL that gives exactly these rights on an EOS project space (a folder that only its creator can write in, under a parent where every CMS member can create one, with deletion by the service account), because the site does not exist yet. Whether files that a user writes count against the project's quota is also a question for EOS support.
Before the site is announced, test with two ordinary accounts that neither can write, replace or remove anything in
the other's folder: ember --user <the other> FILE.root, ember rm --user <the other> NAME, and plain cp and rm on
the mounted folder. Check also that the cron account can.
Who is looking: whoami¶
The viewer lists a person's own files under My files. To know who is looking, it asks the site for whoami
(relative to the page), and expects {"login": "jsmith"}. It ignores any other answer, or no answer. In that case
the viewer asks the person for their CERN login once, and remembers it in the browser. The site works without
whoami.
Here is a minimal script. It is untested: it needs CGI execution enabled on the webEOS site, and the name of the variable in which the sign-on passes the login has to be checked there.
#!/usr/bin/env python3
# www/cgi-bin/whoami.py: the signed-in person's CERN login, for the viewer's "My files".
import json, os, re
login = os.environ.get("OIDC_CLAIM_cern_upn") or os.environ.get("REMOTE_USER") or ""
login = login.split("@")[0].lower()
print("Content-Type: application/json")
print("Cache-Control: no-store")
print()
print(json.dumps({"login": login} if re.fullmatch(r"[a-z0-9_-]+", login) else {}))
In www/.htaccess, add a rule that serves it at the address the viewer asks for:
RewriteEngine On
RewriteRule ^whoami$ cgi-bin/whoami.py [L]
The cron jobs¶
There are two entries in the service account's acrontab.
service/LXPLUS_SETUP.md sets them up step by
step, with dry runs first.
*/15 * * * * lxplus.cern.ch <service>/code/service/run_sync.sh >> <service>/logs/sync.log 2>&1
20 4 * * * lxplus.cern.ch <service>/code/service/run_clean.sh >> <service>/logs/clean.log 2>&1
- sync (
ember sync), every 15 minutes. For each campaign ofservice/campaigns.json, it converts the input files that have no bundle yet (one per pass by default). Then it rewritesindex.json. A lock keeps two passes apart. A file that fails three times is left alone until you remove its entry fromstate/failures.json. - clean (
ember clean), once a day. It removes: - users' bundles published more than 30 days ago;
- files that a publish left half-copied more than a day ago.
Then it rewrites each user's index, to list exactly the bundles present. It touches nothing outside data/u/, and
nothing there that is a link. ember clean --dry-run lists what it would remove. --days N (1 or more) changes the
30 days.
Shared samples: RelVal campaigns¶
A new user should have something to open without running anything. For this, a campaign can take CMSSW
reconstruction files. With "export" in its entry, sync first makes the NanoAOD of the first max_events events
of each file, as ember FILE.root does: it writes ember_cfg.py from the modules of the CMSSW area that the entry
names, and runs it in a clean environment set up from that area (CMSSW side). Then it converts the
NanoAOD:
{ "name": "relval_CMSSW_20_1_0_pre6", "title": "RelVal CMSSW_20_1_0_pre6",
"inputs": "/eos/cms/store/relval/CMSSW_20_1_0_pre6/RelVal*/GEN-SIM-RECO/*_D121_*/*/*.root",
"xrootd": "root://eoscms.cern.ch/", "profile": "auto", "max_events": 10,
"stem": "wildcards", "stem_parts": [0, 1, 2],
"export": {"cmssw": "/eos/project/c/cms-ember/service/cmssw/CMSSW_20_1_0_pre6", "geometry": "D121"} }
This makes one bundle of 10 events per sample, under the heading "RelVal CMSSW_20_1_0_pre6" in the viewer's list.
A new release is a new entry, with its own CMSSW area: run cmsrel once for it. For the hits of each cluster and
the truth, merge the CMSSW branches into that area and build it, until they are in the release
(CMSSW side).
Unverified: the path pattern is an example, based on the layout of an earlier release. No RelVal file has been through this under acron yet. The export was measured by hand on step3 files: about 20 s for the job to start, then 0.6 to 0.7 s per event (CMSSW side).
Deploying¶
From a clone, on a machine with node, python3 and the site's EOS space mounted, as the service account:
service/deploy.sh # or: EMBER_SITE_DIR=... EMBER_SITE_URL=... EMBER_SERVICE_DIR=... service/deploy.sh
The script:
- builds the viewer and copies it to the site: new assets first, then
index.html, then it removes the old assets. So a visitor never gets a page that points at missing files; - copies
embertools/andservice/tocode/, and records the site incode/site.env; - builds
emberas one file, and puts it inbin/.
Deploying again is how you update the viewer, the converter and the command. Users with an alias to bin/ember get
the new command at once. Bundles already published are not touched. A bundle says how it is drawn, so old bundles
keep working with a new viewer.
A site set up before the project was called Ember keeps working after a deployment. Its bundles end in .evdb:
the viewer opens them, old links still work (also once a bundle was published again as .ember), sync does not
convert their inputs again, and clean expires them. New bundles end in .ember. Three things change: the
environment variables are now EMBER_* (rename the EVD_* ones: the command and deploy.sh stop while an old one
is set without its new name), the command is installed as bin/ember with bin/evd a link to it (so old aliases run
the new command), and the old code/evdtools/ is left in place (remove it). People who installed the command with
pip should run pip uninstall evdtools, or the old evd stays on their PATH.
Quota and retention¶
- A bundle of ttbar is about 0.4 MB per event without pile-up and 5 to 6 MB per event at 200 pile-up, without truth.
With truth it is larger: about 17 MB per event at 200 pile-up, and the ttbar bundles of Ember's earlier export are
0.85 MB and 20 to 30 MB per event.
emberwrites 10 events by default, so a bundle at 200 pile-up is 50 to 300 MB. A hundred users who each keep five of the largest use 150 GB. - Bundles expire 30 days after they were last published.
ember clean --days N, orEMBER_KEEP_DAYSfor the cron job, changes this. - There is no quota per user: one person can fill the space. If that happens, the folder sizes under
data/u/show who. - Shared groups do not expire. A RelVal campaign of 10 events per sample takes about the same space per sample.
- The 1 TB that a project space gets by default is enough to start.
What each visitor costs¶
Each open page reads index.json and the visitor's own index every 30 s while the tab is visible. It makes one
ranged read per event shown (0.4 to 30 MB). The web server only serves files. There is nothing to scale except EOS
and the web front end, which CERN IT runs.