HitMark

The trigger, again, was my son. After every game he would scrub through long at-bat videos to self-analyze, then send those same multi-minute clips to advisors, coaches, and friends just to show the one swing at the end that mattered. The good part of an at-bat is a few seconds long. Everything around finding it and sharing it was manual. HitMark automates that: drop in the video, and it detects every pitch and cuts each one into its own clip, ready to trim, slow down, and send.

Why it all runs on the phone

On-device was the preference from the start, for privacy, and then the practical constraints made it the only sensible answer. Video files are large and the processing is heavy. Uploading a tournament's worth of at-bats to a server would be slow for the user and expensive for me, and that cost would have to come back as a subscription. I wanted the app to be usable for free at some level, and paying for server capacity per video makes a free tier impossible. So everything runs locally. On an iPhone 13, a two minute at-bat takes about 15 seconds to process.

Audio leads, video follows

The obvious approach is to look for the ball in each frame. I tried that first, including BaseballCV, and it does not hold up. A baseball is small and not visually distinct, and in a single frame it is often hard to find even for a person who knows it is there. Per-frame detection produced so many false positives, and ran so slowly, that it was not feasible.

So the architecture inverted. Audio proposes and video disposes. Spectral flux on the audio track catches the transient of contact, the bat ping or the mitt pop, and it catches almost every real candidate: the audio gate measures between 95.6% and 100% recall on my corpus. Video is then a filter rather than a finder. Vision body-pose tracking and a custom-trained TrackNet model run over the audio candidates and throw out the ones that are not pitches.

TrackNet earned its place by being a better version of the frame diffing I was already doing. It is tuned to find ball-shaped areas of movement between frames, which is a far more reliable signal than trying to identify a ball in any single frame. Motion is what distinguishes a pitch. The ball itself mostly is not.

What the numbers actually are

I built up a labeled library of test footage and every change gets measured against it rather than eyeballed. The corpus is currently 117 labeled videos containing 426 pitch events, split into a training set and a 14 video holdout that tuning never sees.

The shipping Quick mode gets 88.5% recall at 85.2% precision on that holdout, F1 0.868, with a median of about 3.9 seconds per video in the harness. Deep mode runs TrackNet only on the audio candidates that Quick already rejected, which buys recall and costs precision: 90.4% recall at 73.4% precision, F1 0.810, at 6.8 seconds median. That tradeoff is the whole reason there are two modes rather than one.

The number I am happiest with is the one nobody asks about. When it does find a pitch, the median error on the moment of contact is 3 milliseconds against ground truth. Detection is the hard part. Once a pitch is found, the cut lands where it should.

Where it still breaks

Sharp noise that is not a bat is the main enemy. A parent yelling a name right before the pitch can look like contact to a flux detector, and cheering does the same. Batting cages are worse, because bats in the neighboring cages are real contact sounds that simply belong to somebody else, and balls hitting the net behind the camera add more.

Camera position matters too. Filming from behind the batter is still not great. The two failure modes a user actually sees are a clip of the batter standing there doing nothing, which is a false positive, and a missed swing, which usually happens when the detector locked onto a voice instead of the bat.

The experiments that failed

I keep a running ledger of experiments. The current round has five ideas in it, all built, all measured, and so far not one of them has earned its way into the shipping configuration. Building motion tracks independently inside each rejected audio candidate lost precision. A scene-axis proposal meant to recover portrait and behind-the-plate angles turned out to be a safe no-op with no demonstrated gain. Selecting one ball flight per audio anchor before deduplication lost precision. Combining all three did not offset the losses. A small learned model trained to replace one of my hand-tuned thresholds failed on the holdout at the only setting that still did its job. That round is not finished and some of it may yet turn into something, but nothing from it ships today.

Earlier rounds went the same way. Two more sophisticated onset detectors did not beat plain spectral flux. Separating the audio into harmonic and percussive parts improved one narrow band of cases and cost fifteen to forty times more compute for it. Ball blob size looked like it should tell me how far away the ball was, and did not. Widening the scan window let unrelated motion poison the direction and deduplication decisions, so pitches that had previously been fine started failing.

Writing that down is the point. Without the harness I would have shipped two or three of those on the strength of a few videos that looked better.

What the AI did, and what it did not

This is the first Jucius Labs project built around a custom-trained vision model, and it was built nights and weekends with heavy AI assistance. The honest division of labor: the model knows far more about computer vision and audio DSP than I do, and it can survey current techniques faster than I could ever make time for. It built the evaluation harness, which is the single most valuable thing to come out of this project, because it is what turns opinions into numbers. It also does most of the iOS work, since I am not an iOS developer.

What it does not do is decide. Five rejected hypotheses is what that looks like in practice: plenty of plausible ideas, implemented quickly, and then killed by measurement. The judgment about what to keep is the part that is still mine, and 30 years of writing software is mostly what that judgment is made of.

Training moved too. An earlier non-TrackNet vision model took over 24 hours on a MacBook Air and was not especially accurate. The current model trains on Kaggle's free tier in a few hours.

Pro came from a hitting coach

The delayed replay loop was my son's hitting coach's idea. He used a separate app for it during lessons, expensive and not very good. Recording in the cage and reviewing the swing seconds later on a loop is obviously useful, so it belonged in the app. Most camera delay apps get the details wrong: no control over resolution or frame rate, broken handling of portrait versus landscape, no front-facing camera support. It is now my son's favorite feature.

Building a highlight reel from several swings looked like a reasonable upsell and turned out to be harder than expected, mostly because video editing and the performance budget fight each other. It works now.

Detection runs entirely on the device. Nothing uploads, and the video never leaves the phone.