Embeddings gave us a simple way to say “this item is close to that one in meaning,” which was a helpful first signal for practice recommendations. On their own they were never enough for me to call the job done—I also needed how the learner was doing: accuracy over time, how long items took, which wrong answers kept appearing.
In one stack I worked in, the pieces looked like FastAPI for services that felt “ML-shaped,” Postgres for relational truth, Redis for short-lived caches of “what we last recommended,” and S3 for heavier assets. Next.js handled the surface learners saw. None of those choices are unique; what mattered was how we combined signals.
What we indexed
We embedded items (and smaller concept tags where it made sense) when content changed or on a schedule. Attempts lived in ordinary tables keyed by learner and item. At recommendation time I typically:
- pulled nearest neighbors in embedding space as a candidate set;
- filtered in application code by mastery, difficulty, and recency;
- sometimes down-ranked formats the learner had not seen yet, depending on product rules.
When vectors lived in Postgres, a neighbor query often resembled:
SELECT id,
1 - (embedding <=> $1::vector) AS similarity
FROM content_item
WHERE embedding IS NOT NULL
ORDER BY embedding <=> $1::vector
LIMIT 25;
The SQL got us candidates; policy in code decided what was pedagogically sensible. I try to keep that distinction clear—cosine distance is not the same as “what this student should see next.”
Drift and model changes
If you swap embedding models without rebuilding vectors, “near” quietly changes. I learned to either re-embed or keep versioned indexes so experiments stayed honest. It is an easy mistake; I made it once and wasted time debugging an A/B test that was not comparable.
Measurement
We cared about outcomes, not only clicks. Click-through told us something; whether practice helped on later items told us more, when we had data. I never felt we had “solved” recommendations—there was always another edge case—but combining geometry with learning signals felt fairer to students than similarity alone.
I offer this as a sketch from one project’s path, in case it helps you design your own checks and balances.