Skip to content

Field notes: porting scaffold_analysis from omnium

October 2026, remake 0.8.5+ (04462f3). The u-be530 RCE analysis was ported from omnium to remake: 29 rules and 13,113 tasks, run with -E slurm on JASMIN LOTUS2. Its outputs were compared against a fresh omnium reference run. These notes list what got in the way and what could make the next port easier. They are a record of one port, not normative docs.

What worked well

  • One command for the whole cluster run. remake run -E slurm submitted 29 dependent arrays. The omnium reference needed a hand-written sharding driver and five chained sbatch stages.
  • Partial reruns. After fixing a handful of analysers, remake run -E slurm -Q "rule in [...]" resubmitted exactly the failed tasks plus the stale tasks of the changed rules. remake info showed failed and stale tasks separately, which made the fix-and-rerun loop easy.
  • Explicit inputs surfaced hidden dependencies that omnium never tracked: files read through cd, side outputs of other analyses, and config files read via a suite directory.
  • Planning speed. remake info on 13k tasks with callable inputs took about 15 s.

Pain points

1. uses keys silently lose to closure variables in rule factories

With ~30 near-identical rules, the rules were generated by factory functions:

def _per_group_rule(name, groups, ...):
    @rule(..., matrix={'group': list(groups)},
          uses={'analyser': ANALYSERS[name], 'groups': GROUPS})   # GROUPS: {name: [expts]}
    def run(inputs, outputs, group):
        run_analyser(..., expts=groups[group])                     # groups: the factory's list!
    return run

Inside run, groups resolved to the factory's parameter (a list of group names), not to the uses value (a dict). Every task of every group rule failed with TypeError: list indices must be integers or slices, not str, but only at run time on the cluster.

  • The existing ScopeWarning only checks a uses key against module globals.
  • Improvement: also warn when a uses key matches a name that is free in the rule function and bound in an enclosing function scope (fn.__code__.co_freevars).
  • Improvement: document a rule-factory pattern (see 2).

2. Rule factories are not documented

Generating rules in a factory works (name= sets the rule name), but the docs don't cover:

  • that every generated rule shares one function source, so code changes are only told apart through uses (here the analyser class);
  • that input/output lambdas created in the factory capture values that change detection can't see (the closure warning in Rules and tasks applies, but it isn't obvious it covers this case);
  • how rules_from_current_module() finds rules held in module-level variables.

Improvement: an ex13_rule_factory.py example and a short section in Rules and tasks.

3. The job environment is easy to get wrong with pixi

The SLURM guide says jobs inherit the submitting shell's environment. With pixi, PYTHONPATH set outside pixi run is silently replaced by the env's activation, so the remakefile imported the wrong checkout of the analysis package:

PYTHONPATH=$PWD pixi run -e remake remake run pipeline.py        # PYTHONPATH overridden
pixi run -e remake env PYTHONPATH=$PWD remake run pipeline.py    # works

Improvements:

  • Mention the pixi run ... env VAR=... remake ... form in SLURM → The job environment.
  • Record sys.executable, sys.path/PYTHONPATH and the remakefile's resolved imports in .remake/submit.sh or the run log, so a mismatch is visible after the fact.
  • A remake check-env (or run -n --check-env) that imports the remakefile in a fresh subprocess with the job's environment would catch this before submitting thousands of tasks.

4. No way to run tasks in an older Python

remake needs Python ≥ 3.10. The reference env (iris 2.4) needs Python 3.7, so the port's "refactor reproduces the reference exactly" step had to use a separate hand-written driver that calls the analysis code directly.

Improvement: a remake export-task <key> (or ls-tasks --json) that writes a task's resolved inputs, outputs and kwargs as JSON. Any interpreter could then run the task body without importing remake.

5. Resubmitting while a SLURM run is still in flight

While one rule's array was still running, it wasn't clear whether a second remake run -E slurm would plan those running tasks as pending and submit them again. I restricted the second run with -Q to the rules that weren't running.

Improvement: document the behaviour. Better still, have the planner skip tasks that have a live entry in .remake/tasks/running/, or warn about them.

6. Finding the error for each failed task

Errors are split between .remake/tasks/log/<rule>/ (task-level tracebacks) and .remake/slurm/output/<rule>/*.err (failures before the task starts, e.g. import errors). With 64 failures across 11 rules, I ended up grepping both.

Improvement: remake info --failed (or remake failures) printing one line per failed task: the rule, task key and last line of the traceback, plus the log path.

7. Smaller items

  • Zero-padded matrix values. Templates like atmos.{runid:03d}.pp1.nc may or may not be supported (format specs in placeholders). I used string matrix values ('240') to be safe. The docs could say either way.
  • Input order. When a task passes its inputs to code that expects an ordered list, the docs don't say whether the inputs dict keeps its order through planning and serialisation. I used sortable keys (in000, in001, ...). A sentence in Rules and tasks would settle it.
  • Shared side outputs. Some analyses write one file that every task of the rule overwrites (a per-rule summary figure). Declaring it as an output triggers the "shared outputs" fallback; not declaring it means it isn't tracked. A short note on how to handle these (a separate fan-in rule, or leave undeclared) would help.
  • Reused SLURM job IDs on LOTUS2. sacct -j <id> returned an unrelated job from January with the same ID. Anything in remake that queries sacct by job ID should also pass -S (the submission time) to avoid misattributing old failures.