Laya service

Local Python service for Convai Innovations Laya. Laya is a non-autoregressive decision model: it evaluates typed questions over text or structured state and returns answers, probabilities, confidence, and usage metadata. It is not an Ollama model and does not generate text.

The service supports two bundled checkpoints:

• english — the English ModernBERT checkpoint, selected by default.
• multilingual — the bundled 100+ language checkpoint.

Credits

This service uses Laya by Convai Innovations. Please see and credit the upstream project:

• Laya model on Hugging Face
• Laya source repository on GitHub
• Convai Innovations on Hugging Face
• Laya developer profile on GitHub

Quick start

Requirements: Python 3.13+, uv, and a working NVIDIA driver if using CUDA. CPU mode is also supported.

cp .env.example .env       # once; edit the LAN CIDR and preferred mode/model
uv sync
uv run python scripts/quickstart.py

Run the LAN API with the same .env configuration:

./run.sh

The first model load downloads the selected checkpoint into the configured Hugging Face cache. The API uses one worker by default because every worker owns its own model copy.

Configuration

All supported settings are documented in .env.example. Python loads the project-root .env automatically using python-dotenv; values already present in the process environment take precedence. run.sh is only a thin launcher and does not duplicate configuration parsing, so the same arrangement works in a future container with externally injected variables.

Common settings:

HF_HOME=.cache/huggingface
LAYA_ALLOWED_CIDRS=192.168.178.0/24,127.0.0.1/32
LAYA_DEVICE=cpu
LAYA_MODEL=english
LAYA_HOST=0.0.0.0
LAYA_PORT=8000
LAYA_WORKERS=1

Use LAYA_DEVICE=cuda for the RTX GPU or LAYA_DEVICE=cpu for CPU inference. CUDA is the Python runtime default when the variable is absent; this checkout’s local .env intentionally selects CPU. Use LAYA_MODEL=multilingual to select the multilingual checkpoint. Selection is explicit; the service does not automatically route between checkpoints.

LAYA_ALLOWED_CIDRS is mandatory for the API and accepts comma-separated IPv4 CIDRs. Requests are checked against the direct socket peer; forwarded-IP headers are not trusted.

Model management

Refresh both English and multilingual checkpoint files from Hugging Face:

uv run scripts/update_models.py

The updater uses HF_HOME, checks the Hub revision, and downloads only the files belonging to each checkpoint. Set HF_TOKEN in .env when authenticated Hub access or higher download limits are needed. Model files and .env are ignored by Git.

Direct quickstart

The model-card example can run against either mode/model selected in .env:

uv run python scripts/quickstart.py

To temporarily override configuration without editing .env:

LAYA_DEVICE=cuda LAYA_MODEL=multilingual uv run python scripts/quickstart.py

The script prints runtime diagnostics and evaluates choice, score, and noul questions over a sample billing email.

HTTP API

GET /health returns readiness, the selected mode/model, and device diagnostics:

curl http://127.0.0.1:8000/health

POST /v1/predict accepts a string, JSON object, or JSON list as state, plus at least one typed question:

curl -X POST http://127.0.0.1:8000/v1/predict \
  -H 'content-type: application/json' \
  --data '{
    "state": "My card was charged twice; please refund the duplicate.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this?",
        "criteria": {
          "billing": "payments and refunds",
          "technical": "bugs and outages",
          "other": "everything else"
        }
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this?",
        "criteria": ["not urgent", "soon", "critical"]
      },
      "refund_requested": {
        "type": "noul",
        "instructions": "Does the user request a refund?"
      }
    }
  }'

Run the model capability probe with the English catalog (20 everyday scenarios):

uv run python -m laya_service.model_probe http://127.0.0.1:8000 --catalog english

Use the multilingual catalog to exercise the multilingual checkpoint with 20 varied German scenarios:

uv run python -m laya_service.model_probe http://127.0.0.1:8000 --catalog multilingual

scripts/api_demo.py remains a thin compatibility launcher for the same module. The probe prints one timed status line per request, updates an in-place progress line while a request is running, and uses terminal colors for section headings and request failures. Each run writes a timestamped plain-text log in the current directory. Pass --log path/to/probe.log to choose the path; the log contains every request and response as indented JSON and cumulative runtime.

After each probe line, the client prints an indented breakdown for every question. Choice questions list every option probability, noul questions list the true/false probabilities, and score questions list the score and each level probability. The detailed lines show the model output without applying pass/fail thresholds. The complete original state and question definitions are printed immediately above each question's indented result block, with colored section headers for state and questions.

GPU and CPU behavior

CUDA mode resolves to cuda:0, reports GPU name/VRAM, and rejects Laya’s silent CPU fallback. CPU mode is intentional and slower but uses the same API and checkpoint. On this machine, the verified GPU is an NVIDIA RTX 2000 Ada Laptop GPU with approximately 8 GiB VRAM.

Host-level CUDA diagnostics:

nvidia-smi
uv run python -c 'import torch; print(torch.cuda.is_available(), torch.version.cuda)'

Development

uv sync
uv run pytest
uv run ruff check .
uv run ruff format --check .
bash -n run.sh
uv lock --check

See AGENTS.md for the architecture, environment contract, API details, testing strategy, container guidance, and conventions for future changes.