Appverse Reviewer Checklist

Related Docs: Appverse Contributor Guide | Appverse Security Rubric | Appverse Best Practices | Appverse README Template

A guide for reviewers evaluating whether a new app should be included in the Appverse catalog.

Overview

The goal of review is to ensure that apps in the catalog are useful, deployable, and maintainable by the broader HPC community. We're not looking for perfection — we're looking for apps that meet a quality bar and don't create confusion in the catalog.

Decision Framework

Should This App Exist in the Catalog?

Ask these questions first, before evaluating quality:

Duplicate Check

If the software already has apps in the catalog:

• Same app, different site config → Reject. Encourage the contributor to document their configuration in the existing app or contribute changes upstream.
• Same app, forked with minor changes → Reject unless the fork adds significant new capability. Encourage contributing back upstream.
• Same software, meaningfully different approach → Accept. Examples: containerized vs. module-based, GPU vs. CPU, different execution model (e.g., AlphaFold 2 vs. AlphaFold 3 support).

Document the rationale either way — the review report has a duplicate-check rationale field under "Not checked" for this; fill it in before pasting the feedback into the issue or email.

Reading the catalog without a login

The catalog is a Drupal site with JSON:API enabled and readable anonymously, so the Software entry, the vocabularies and the published app list can all be checked from the command line — no reviewer account needed.

Two things will make these look like an unreachable host if you get them wrong. Follow redirects (-L), or the request returns 000. And percent-encode the query brackets as %5B/%5D — a URL with literal [ is dropped before it reaches Drupal and yields an empty response with no status code at all.

# Does a Software entry exist for this app's `software` value?
curl -sL -H 'Accept: application/vnd.api+json' \
  'https://openondemand.connectci.org/jsonapi/node/appverse_software?filter%5Btitle%5D=API%20Access&fields%5Bnode--appverse_software%5D=title'

# The implementation-tags vocabulary (matching is case-insensitive)
curl -sL -H 'Accept: application/vnd.api+json' \
  'https://openondemand.connectci.org/jsonapi/taxonomy_term/appverse_implementation_tags?fields%5Btaxonomy_term--appverse_implementation_tags%5D=name&page%5Blimit%5D=50'

# Published apps, for the duplicate check
curl -sL -H 'Accept: application/vnd.api+json' \
  'https://openondemand.connectci.org/jsonapi/node/appverse_app?fields%5Bnode--appverse_app%5D=title&page%5Blimit%5D=100'

Sibling resources on the same API: taxonomy_term--appverse_app_type, taxonomy_term--appverse_license, node--appverse_repo. GET /jsonapi lists everything available.

Writing still needs a login — creating a Software entry, moderating an app. Reading does not, so report these checks as performed rather than deferred.

Software Entry Check

The app's software value must match a Software entry in the catalog or the app won't be listed. Software is its own catalog node type, created separately from the app — the app form only references an existing Software entry, it cannot create one. Query for it with the command above. If there is no matching entry, it is the reviewer's decision, not an automatic failure:

• Software should exist in the catalog → create the Software entry first using the Add Appverse Software form (reviewer login required), then the app can reference it and list.
• Typo, or maps to an existing entry under a different name → ask the contributor to correct the software value.
• Not appropriate for the catalog → Request changes, with the reason.

Declared vs. Inferred Repos and Monorepos

A repo with a root appverse.yml is declared: the contributor states how their app(s) appear in the catalog. A repo with only a root manifest.yml is inferred: the catalog derives a single app from that file plus GitHub metadata.

A declared repo whose appverse.yml has an apps: list is a Monorepo. Review every entry in apps[] as its own app — each gets its own structure, security, and quality assessment, and its own decision. Per-app fields resolve with this precedence:

1. Inline in the root appverse.yml apps[] entry (highest)
2. <subpath>/appverse.yml
3. <subpath>/manifest.yml (name/description fallback only)

Directories listed in shared_paths are shared code: review them once at repo level and include them in the security review. Decisions can differ per app — e.g., accept three apps and request changes on a fourth.

Required Criteria

All of these must be met for inclusion. These are pass/fail.

1. Repository Structure

Every repo, regardless of shape:

Inferred repos (root manifest.yml, no appverse.yml):

Declared repos (root appverse.yml), checked per app after field-precedence resolution:

For Batch Connect Apps: standard OOD structure with expected files (form.yml or form.yml.erb, submit.yml.erb, template/). OOD renders form.yml.erb at request time, so an app shipping only the .erb variant is complete; shipping both is unusual and worth a look

2. Documentation Minimum

The README should follow the Appverse README Template structure. At minimum, it must answer these questions for an HPC admin who has never seen the app:

Red flags:

• README is still the unfilled template (placeholder text like "Key feature 1" still present)
• README only contains a project name and/or contact email
• README references a different institution's paths without noting they need to change
• No Configuration section — deployers can't tell what to customize

3. Security Concerns

Security review follows the dedicated Appverse Security Rubric: build a capability profile for the app's type, run the pattern checks, and classify findings under the OODT taxonomy. The three legacy spot-checks (committed credentials, disabled security without justification, user input reaching shell commands) are all covered by the rubric's pattern checks.

4. Basic Functionality

Quality Criteria

These are evaluated on a scale. An app doesn't need to be perfect, but the more boxes checked, the better.

Documentation Quality

Target for inclusion: Adequate or above.

Configuration Portability

Target for inclusion: Partially portable or above. If "Not portable," ask the contributor to centralize configuration before accepting.

Code Quality

Target for inclusion: At least error handling and input validation. Other items are improvement suggestions.

Maintenance Signals

Repo-level signals about the project's health.

Target for inclusion: Active within 12 months. For brand-new apps, waive the history requirement but flag for follow-up review in 6 months.

The other signals — tagged releases, issue responsiveness, contributors, CHANGELOG, CI — are good-practice indicators, not requirements. There is no requirement to cut releases or use versioning; a release is a positive signal that the team follows good practices, and its absence is worth a suggestion, never a failure. CI is the weakest of these for an app repo: most Appverse apps are config-and-template repos with little to test beyond YAML validity and shell/ERB lint, which Appverse Review already checks — so weight a missing CI workflow low.

Review Process

Where to Find Pending Submissions

Apps awaiting review appear in the Manage Appverse Apps view in Drupal: openondemand.connectci.org/appverse/manage-apps. The view shows the submitter's name and email so you can follow up with questions, the moderation state, and a link to edit the app node.

Most of the checks below are performed by Appverse Review, which produces an evidence-backed report covering structure, security, quality, and maintenance, with file:line findings and a recommended decision. The reviewer receives this report, verifies its findings, and makes the call — the review recommends, a human decides.

Step 1: Start from the review report

Begin with the Appverse Review report for the submitted repo. Note the commit it was run against — the report pins a commit SHA, and its file:line findings are only valid for that commit. If the repo has moved on since, get a fresh report before continuing.

Step 2: Verify the findings

Read the report against the repo and confirm its findings hold, rather than taking them on trust. Spot-check that:

1. Required files and metadata are as the report states — appverse.yml or manifest.yml, README.md, LICENSE, standard OOD structure.
2. Security findings are real and correctly rated — open a few and check the cited file:line actually shows the flagged pattern.
3. Quality ratings and any correctness-&-polish findings match what you see — documentation level, portability, and any copy-paste artifacts or typos.
4. Maintenance signals are current — last commit, releases, CI, CHANGELOG.
5. The duplicate check is settled — the review cannot see the catalog, so this is the reviewer's to confirm (see Duplicate Check above).

Trust the report's structure but verify its substance; if a finding does not hold, correct it before it reaches the contributor.

Step 3: Decision

Providing Feedback

When requesting changes or suggesting improvements, be specific:

Good feedback:

The README lists prerequisites but doesn't include installation steps. Please add a section showing how to clone and deploy the app (see ProteinStructure-OOD for an example).

Poor feedback:

README needs work.

Reference the Best Practices Guide for specific recommendations to share with contributors.

Common Issues Found in Real Apps

Based on analysis of apps currently in the Appverse:

Appendix: Review Template

Copy this template when reviewing a new app submission:

## App Review: [App Name]

**Repository:** [URL]
**Reviewed commit:** [SHA] ([date])
**Reviewer:** [Name]
**Date:** [Date]

### Required Criteria
- [ ] Required metadata fields for the repo shape (see Repository Structure)
- [ ] README.md (substantive)
- [ ] LICENSE (open source)
- [ ] Standard OOD app structure

For Monorepos: repeat the per-app criteria and decision for each entry in `apps[]`.

### Security
- Findings classified under the OODT taxonomy, with severity and file:line evidence

### Quality Assessment
- Documentation: [Minimal / Adequate / Strong / Exemplary]
- Portability: [Not portable / Partially portable / Portable]
- Code quality and correctness-&-polish findings, with file:line evidence

### Maintenance Signals
- [Active / Moderate / Inactive] — last commit, releases, CI, CHANGELOG

### Catalog Checks
- [ ] Duplicate check against the existing catalog — [outcome and rationale]
- [ ] `software` value matches a catalog Software entry (see Software Entry Check — create the entry if the software should exist)
- [ ] `app_type` and `implementation_tags` are in the catalog vocabularies

Query these against the public JSON:API — see "Reading the catalog without a
login". Record what each returned. If one could not be run, say which and why,
rather than leaving it unmarked.

### Decision: [Accept / Accept with suggestions / Request changes / Reject]
- If any catalog check above could not be run, Accept is conditional on it

### Feedback
[Specific items to address or improve — every item must already appear above]