Installing Jev on a Two-Node DGX Spark Cluster

 A TECHNOCRAT'S DISCERNMENT  ·  BUILD LOG

A hands-on setup guide. Nemotron as the generator, Jev as the decision layer, TypeScript holding it together.

This is the guide I wish I had on Saturday morning. We are going to stand up two NVIDIA DGX Spark units as a single cluster, serve a Nemotron model across both of them as the heavy generator, and then put Jev in front as the fast decision layer that decides what the generator ever has to see. One box would work for a smaller model. Two boxes, joined by a single cable, give us 256 GB of unified memory and the room to run a real generator at home.

One point worth stating up front, because it changes how you read every step below. Nemotron runs locally on your cluster. Jev does not. Jev is TypeSafe AI's hosted System One model, and you reach it over HTTPS through the jev-code CLI or the TypeSafe SDK. So what you are installing on the cluster is the local generator, the Jev client, and the TypeScript harness that wires the two together. The decisions leave the building. The generation stays home. That split is the whole point.

What you need

Two DGX Spark units, each with a GB10 and 128 GB of unified memory, running DGX OS. A single QSFP cable to connect them directly, no switch required. Each Spark ships with ConnectX-7 NICs that carry 200 Gbps of RoCE, which is the fabric the cluster runs on. On the software side you want Docker with the NVIDIA Container Runtime working on both nodes, Node.js 22 or newer for the Jev tooling, and a TypeSafe API key. Set the same username and password on both machines before you start, because the cluster scripts lean on SSH between them.

Step 1  ·  Prepare both Sparks

Update both machines and confirm the GPU stack is alive on each before you try to make them talk to each other. Run this on spark-a and again on spark-b.

sudo apt update && sudo apt -y upgrade

nvidia-smi            # confirm the GB10 is visible

docker info | grep -i runtime   # expect: nvidia

Give the two machines names you will not mix up. I use spark-a for the head node and spark-b for the worker, and I add both to /etc/hosts on each side so the scripts and I are talking about the same boxes.

Step 2  ·  Verify the 200 Gbps link

Connect the QSFP cable between the two ConnectX-7 ports, then prove the link is actually running at 200 Gbps before you build anything on top of it. A slow or misconfigured fabric will not error out cleanly later. It will just make everything mysteriously terrible.

sudo ethtool enp1s0f1np1 | grep Speed     # expect: 200000Mb/s

Give the two interfaces link-local addresses so they can find each other. Create /etc/netplan/40-cx7.yaml on both Sparks with the same contents, then apply it.

network:

  version: 2

  ethernets:

    enp1s0f1np1:

      link-local: [ ipv4 ]

    enP2p1s0f1np1:

      link-local: [ ipv4 ]

 

sudo netplan apply

Now set up passwordless SSH from the head to the worker over that fabric, because the cluster launch drives the worker over SSH.

ssh-keygen -t ed25519

ssh-copy-id -i ~/.ssh/id_ed25519.pub <user>@<worker-ib-ip>

ssh <user>@<worker-ib-ip> hostname     # should print spark-b, no password

If you would rather not hand-roll any of this, NVIDIA Sync ships a cluster assistant that walks the ConnectX-7 topology, plans the addresses, validates bandwidth, and exchanges keys for you. I did it by hand once to understand it, then let Sync do it every time after.

Both nodes reporting a 200 Gbps link with active RoCE ports. Representative capture.

Step 3  ·  Bring up Nemotron on the cluster

The generator runs as a Ray cluster with vLLM on top, tensor-parallel across both Sparks. The simplest path I found is the community vLLM-on-DGX-Spark project, which handles the head and worker containers, the model download, and the join. Clone it on the head node and set your worker details.

git clone https://github.com/mark-ramsey-ri/vllm-dgx-spark

cd vllm-dgx-spark

source ./setup-env.sh          # or edit config.local.env by hand

Edit config.local.env so the head knows where the worker is and how to split the model. Two GPUs means tensor-parallel size two. For the model, point it at the Nemotron build you want to serve.

WORKER_HOST="<worker-ethernet-ip>"

WORKER_IB_IP="<worker-infiniband-ip>"

WORKER_USER="<ssh-username>"

TENSOR_PARALLEL="2"

MODEL="Nemotron-3-Super-120B-A12B-NVFP4"

Nemotron-3-Super-120B is a mixture-of-experts model, 120 billion parameters total with about 12 billion active per token, which is exactly the kind of thing the 256 GB of pooled memory was made for. Launch it in FP4 with the marlin backend, and pass the InfiniBand device flags to both containers so NCCL uses the fast fabric instead of quietly falling back to TCP.

# vLLM launch flags used by the cluster script

--tensor-parallel-size 2 --pipeline-parallel-size 1 \

--quantization fp4 --moe-backend marlin

 

# container flags on BOTH nodes (keeps NCCL on the QSFP fabric)

--device=/dev/infiniband --cap-add=IPC_LOCK --ulimit memlock=-1:-1

 

# environment on BOTH nodes

VLLM_NVFP4_GEMM_BACKEND=marlin

VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm

VLLM_USE_FLASHINFER_MOE_FP4=0

VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

Start the cluster from the head Spark and wait for the join. On a two-Spark setup this took me two to five minutes end to end, most of it model movement.

./start_cluster.sh

 

curl -s localhost:8000/health

curl -s localhost:8000/v1/models

The cluster up, Nemotron served on the OpenAI-compatible endpoint across two nodes. Representative capture.

You now have an OpenAI-compatible endpoint at localhost:8000 backed by both Sparks. That is your System Two. On to the fast half.

Step 4  ·  Install jev-code and set the key

The Jev tooling installs with a single script. Run it on the head node, put the binary on your path, and hand it your TypeSafe key. The key can live in an environment variable, in a local .env file, or you can log in interactively on first run.

curl -fsSL https://raw.githubusercontent.com/rhighs/jev-code/main/install.sh | bash

export PATH="$HOME/.local/bin:$PATH"

 

export TYPESAFE_API_KEY=sk_live_your_key_here   # or: jev-code login

jev-code --version

The key is stored at ~/.config/jev-code/config.json with 0600 permissions. You can prove the whole path is wired with a single decision from the shell, no code required.

jev-code decide "is this a refund request?" \

  --state "I want my money back for order 5512" --choices yes,no

# -> yes   p=0.97   (142 ms)

jev-code installed, key accepted, and a first decision returned in under 150 ms. Representative capture.

Step 5  ·  Run jev-code interactively

Before wiring anything, spend ten minutes watching jev-code build code. It does not write text and then hope it parses. It selects productions from a real grammar one decision at a time, so every line is a valid node in the syntax tree before it is written. Start a session and give it a task.

jev-code

# then, at the prompt:

> build a Python module that computes mean, median, and stdev

The screen splits. On the left a decision strip shows each production Jev selected and how confident it was. On the right the file grows in real time. You accept or reject each tool call, per invocation or per session. Watching a program assemble without ever once emitting a syntax error is a small strange pleasure, and it tells you a lot about what this model is for.

The jev-code session: decision strip with confidences on the left, the live file on the right. Representative capture.

Step 6  ·  Wire the harness in TypeScript

Here is the pattern that matters. Every request hits Jev first. Jev classifies it, scores it, and decides whether it is simple enough to resolve on its own or hard enough to deserve the generator. Only the genuinely ambiguous cases ever wake Nemotron. The rest are routed, scored, and filed for a fraction of a cent.

The shape below is TypeScript against the Jev SDK, with Nemotron reached through its OpenAI-compatible endpoint. Confirm the exact SDK package name against the TypeSafe SDK docs, since the Python package (typesafe-sdk) is the one documented in full at the time of writing. The logic is identical in either language.

import { Jev, Noul, Score, Choice } from "@typesafe-ai/jev";

import OpenAI from "openai";

 

const jev = new Jev({ apiKey: process.env.TYPESAFE_API_KEY });

const nemotron = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "local" });

 

async function handle(request: string) {

  const d = await jev.systemOne({

    state: request,

    questions: {

      needsGeneration: Noul("does this need free-form generation?"),

      risk: Score("operational risk", ["low","medium","high"]),

      route: Choice("which queue?", ["auto","review","human"]),

    },

  });

 

  if (d.risk.level === "high") return escalateToHuman(request, d);

  if (!d.needsGeneration.value) return fileDirectly(request, d);   // no generator call

 

  // only now do we spend a Nemotron token

  return nemotron.chat.completions.create({

    model: "Nemotron-3-Super-120B-A12B-NVFP4",

    messages: [{ role: "user", content: request }],

  });

}

Run it with tsx while you iterate, then compile for anything you keep. The first time you see the generator stay quiet on nine requests out of ten, the economics of the whole thing click into place.

Two use cases worth trying first

Document intake routing

The unglamorous front door of half the enterprise. Every incoming document gets a Choice for its type, a Score for completeness, and a Noul for whether a human needs to look. Only the ambiguous ones reach Nemotron. In my sandbox, feeding a folder of mixed PDFs through the gate, Jev handled the overwhelming majority and the generator was woken only for the genuinely unclear items. The point is not the accuracy on any one document. It is that you stop rationing the checks, because at this price you can afford to ask every question about every document.

Intake routing: Jev gates the batch, Nemotron sees only the ambiguous few. Representative capture.

PR risk gate in CI

This is the one I care about, given how I spend my days. Before any expensive agent run or human review, Jev scores each pull request for risk. Does it touch auth or secrets. Does it ship tests. Is it reversible. Low-risk changes sail through, and only the risky ones pull in Nemotron and a human sign-off. The gate answers in about a hundred milliseconds, so you can put it on every push without anyone noticing the cost.

A risky PR flagged and escalated, the rest waved through. Representative capture.

A few things that will save you an afternoon

If NCCL falls back to TCP your throughput collapses and nothing tells you why. The InfiniBand device flags on both containers are what keep it on the fabric, so do not skip them. If the worker drops mid-run or one GPU pins at 100 percent forever, check that both nodes really did come up with tensor-parallel size two and that the model actually finished syncing to the worker before serving started. Keep the marlin backend for the FP4 GEMMs. And remember the token budget on the Jev side, roughly 64,000 tokens shared across your state and questions, so trim the state you send rather than pasting whole documents into it.

That is the whole rig. Two boxes, one cable, a generator that stays home and a decision layer that costs almost nothing.

Serve the reasoning. Rent the decisions. Watch where the work actually goes.