Python client for benchmarking and validating ML models on real devices via RunLocal API.

🎯 Key Features

• ⚡ Real Hardware Testing - No simulators or emulators. Test on real devices maintained in our devices lab
• 🌍 Cross-Platform Coverage - Access MacBooks, iPhones, iPads, Android, and Windows devices from a single API
• 🔧 Multiple ML Formats - Support for CoreML, ONNX, OpenVINO, TensorFlow Lite, and GGUF models. More frameworks coming soon.
• 📊 Detailed Metrics - Measure inference time, memory usage, and per-layer performance data
• 🚦 CI/CD Ready - Integrate performance and accuracy testing into your deployment pipeline

🔍 Evaluate Results

All benchmarks performed through the python client can be evaluated on the web platform by logging into your account. Check out our public demo for comprehensive benchmark evaluation across different devices and model formats.

🛠 Installation

pip install runlocal-hub

Development Installation

For development or to install from source:

git clone https://github.com/neuralize-ai/runlocal_hub.git
cd runlocal_hub
pip install -e .

🔑 Authentication

Get your API key from the RunLocal dashboard:

1. Log in to RunLocal
2. Click your avatar → User Settings
3. Navigate to "API Keys"
4. Click "Create New API Key"
5. Save your key securely

export RUNLOCAL_API_KEY=<your_api_key>

🕹 Usage Guide

Simple Benchmark

from runlocal_hub import RunLocalClient, display_benchmark_results

client = RunLocalClient()

# Benchmark on any available device
result = client.benchmark("model.mlpackage")
display_benchmark_results(results)

Device Filtering

Target specific devices with intuitive filters:

from runlocal_hub import DeviceFilters, RunLocalClient

client = RunLocalClient()

# High-end MacBooks with M-series chips
mac_filters = DeviceFilters(
    device_name="MacBook",
    soc="Apple M",        # Matches M1, M2, M3, etc.
    ram_min=16,           # At least 16GB RAM
    year_min=2021         # Recent models only
)

# Latest iPhones with Neural Engine
iphone_filters = DeviceFilters(
    device_name="iPhone",
    year_min=2022,
    compute_units=["CPU_AND_NE"]
)

# Run benchmarks
results = client.benchmark(
    "model.mlpackage",
    device_filters=[mac_filters, iphone_filters],
    count=None  # Use all matching devices
)

🧮 Running Predictions

Test your model with real inputs:

import numpy as np

# Prepare input
image = np.random.rand(1, 3, 224, 224).astype(np.float32)
inputs = {"image": image}

# Run prediction on iPhone
outputs = client.predict(
    inputs=inputs,
    model_path="model.mlpackage",
    device_filters=DeviceFilters(device_name="iPhone 15", compute_units=["CPU_AND_NE"])
)

tensors = outputs["CPU_AND_NE"]
for name, tensor in tensors.items():
    print(f"  {name}: {tensor.shape} ({tensor.dtype})")
    print(f"  First values: {tensor.flatten()[:5]}")

📚 Examples

Check out the example scripts:

• bench_example.py - Simple benchmarking example
• predict_example.py - Prediction with custom inputs, serialised outputs

💠 Supported Formats

Format	Extension	Platforms
CoreML	.mlpackage/.mlmodel	macOS, iOS
ONNX	.onnx	Windows, MacOS
OpenVINO	directory (.xml + .bin)	Windows (Intel)
TensorFlow Lite	.tflite	Android
GGUF	.gguf	All platforma

More frameworks coming soon.

📜 License

This project is licensed under the MIT License - see the LICENSE.txt file for details.