VLM AI Model Development

Project Title: Fine-tune a Vision-Language Model to read vital signs from patient monitor photos OVERVIEW We are a healthcare technology startup building a remote patient-monitoring product for hospitals. We need an AI model that looks at a photo of any patient monitor (NICU / ICU bedside monitor) and extracts the vital signs as structured data. The model must work across DIFFERENT monitor brands and layouts (Philips, GE, Drager, Mindray, Nihon Kohden, SLE, etc.) — not just one fixed brand. It must understand that different labels mean the same vital (e.g. "HR", "PR", "Pulse", "Heart Rate" all mean heart rate). This is a semantic understanding problem, NOT a simple OCR or bounding-box detection task. Please do not propose Roboflow / YOLO / Tesseract-only solutions — they cannot generalize across unseen layouts. WHAT THE MODEL MUST DO - Input: one image of a patient monitor (often in a cluttered real-world hospital scene, with staff/equipment in frame) - Output: clean JSON, e.g. { "hr": 142, "spo2": 98, "rr": 45, "bp_sys": 70, "bp_dia": 40, "temp": 36.8 } - Use null for any vital not visible - Must be robust to different fonts, colors, screen layouts, glare and angles SCOPE OF WORK 1. Fine-tune an open-source VLM. Our preferred base model is Qwen2.5-VL-7B (Apache 2.0 license). You may propose an alternative (e.g. InternVL2) only with strong justification, but Qwen2.5-VL is our default choice. 2. Build a data pipeline: we will provide raw monitor images; help us set up an auto-labeling step (using an external vision API to generate first-pass labels) plus a human-verification workflow. 3. Run the fine-tune using LoRA/QLoRA (free GPU environments like Kaggle/Colab are acceptable — keep training cost near zero). 4. Export the final model to GGUF format so it can run locally via Ollama. 5. Deliver a deployment guide so the model runs offline on BOTH macOS (Apple Silicon) AND Windows machines — no internet and no cloud cost. 6. Provide an accuracy report on a held-out test set. CONSTRAINTS (IMPORTANT) - Final model must run OFFLINE and CROSS-PLATFORM — it must work on both macOS (Apple Silicon) and Windows PCs (with or without an NVIDIA GPU). Patient data cannot leave the hospital — privacy requirement. - Target hardware is a mid-range machine with 16GB RAM. Inference speed of a few seconds per image is acceptable. - License must allow commercial use (Apache 2.0 / MIT preferred). No models with research-only licenses. (This is one reason we prefer Qwen2.5-VL-7B, which is Apache 2.0.) - Inference cost after deployment must be zero (self-hosted only). DELIVERABLES - Fine-tuned model weights + exported GGUF file - Training/fine-tuning code (documented, reproducible) - Auto-labeling + data-prep scripts - Deployment instructions for BOTH macOS and Windows (via Ollama) - Accuracy evaluation report IDEAL SKILLS AND EXPERIENCE - Proven experience fine-tuning Vision-Language Models (Qwen-VL strongly preferred; also InternVL, LLaVA, PaliGemma, etc.) - Strong Python; PyTorch; Hugging Face Transformers; PEFT/LoRA; Unsloth a plus - Experience exporting models to GGUF and running with Ollama / [login to view URL] on both Mac and Windows - Comfortable with QLoRA on limited/free GPU resources - Bonus: prior medical / OCR / document-AI projects TO APPLY Please share: 1. Your proposed approach in 3-4 lines 2. Rough timeline and milestone breakdown We have a starter dataset of real monitor photos ready to share with shortlisted candidates.

Project ID: 40551078