Grading Myself Against Hindsight: The Decision Log in My Fantasy Football Assistant

This whole project started with me looking back. In Part 1, I checked my league's championship history to prove fantasy football was mostly luck, and the data proved me wrong: the managers who know the players win most of the time. In Part 2, I covered how the app scores players and how I stopped the AI from recommending an injured receiver.

This post is about the feature that ties it all together, and honestly the one that has taught me the most: the Decision Log. It's the same "look back at the results" exercise that started this project, except it runs on my own decisions, every week.

One note before we dig in: each league in the app keeps its own Decision Log, and everything in this post comes from my keeper league (the one where I'm rebuilding, and where the McMillan suggestion from Part 2 came from).

Fair warning: this post is not a victory lap. The Decision Log's whole job is to show me where I'm wrong, and it's been very good at its job.

Your Memory Is a Bad Scorekeeper

Here's how I used to evaluate my fantasy moves: I didn't. A pickup that worked out felt like a smart call. A pickup that flopped was bad luck, or the coach's fault, or I'd forgotten I made it at all. Two months later, I couldn't have told you why I dropped someone, let alone what I knew at the time.

That's the problem with hindsight. It's not that we can't judge past decisions, it's that we judge them with information we didn't have when we made them, and we quietly rewrite what we were thinking. The only fix is to write things down at the moment of the decision and then compare later.

So that's what the Decision Log does.

Freeze "Then," Keep "Now" Live

Every trade, waiver claim, add, and drop gets imported from ESPN automatically by the nightly sync (or on demand with a Sync button). For every player in a move, the app records two "then" values:

• PVS at the time of the move, from the nearest daily snapshot.
• Fantasy points at the time of the move, backfilled from ESPN's weekly scoring once that week's data is available.

Those "then" values are frozen. They are never recalculated, because the whole point is to compare against them. If they moved every time the scoring changed, "then vs. now" would just be "now vs. now."

The "now" values are the opposite: computed fresh every time I open the page.

The trickiest part was the early days. When I first started using the app, there wasn't any PVS history yet for moves that happened before the first snapshot. Here's how the app picks a "then" value:

def _pvs_at(session, player_id: int, league_id: int, when: dt.datetime, mode: str):
    base = session.query(PVSScore).filter_by(player_id=player_id, league_id=league_id, mode=mode)
    before = base.filter(PVSScore.computed_at <= when).order_by(PVSScore.computed_at.desc()).first()
    if before is not None:
        return before.pvs, False
    after = base.filter(PVSScore.computed_at > when).order_by(PVSScore.computed_at.asc()).first()
    if after is not None:
        return after.pvs, True
    return None, False

It uses the latest snapshot at or before the move. If none exists, it falls back to the earliest snapshot after the move and flags it as an estimate (shown with a ~ in the UI and (est.) in the export). If there's nothing at all, it returns None and the UI shows a blank, never a fake zero. After the McMillan episode in Part 2, I'm a lot more suspicious of values that quietly fill themselves in.

Two Kinds of Hindsight

Each move shows two comparisons side by side, because "was this a good move?" really has two answers:

• PVS then vs. now is a market read. Did the rankings sites' opinion of this player move in my favor?
• Points then vs. now is a production read. Did he actually score?

They disagree more often than you'd think. Here's one waiver claim from my log, where I picked up Caleb Douglas and dropped Wan'Dale Robinson:

By market value, I won that move. By actual points, the guy I dropped outscored the guy I added by about 24 points. Neither number alone tells the full story, which is exactly why the app shows both.

Points also cover something PVS can't. Neither ranking site scores kickers or team defenses, so those moves have no PVS at all. But the points side still works, and it turns out some of my best moves of the season were there: claiming the Vikings D/ST (24 → 60 points) and swapping kickers to Spencer Shrader (32 → 63 points).

Grading Myself

The data is only half of it. The other half is me.

Every move can get an A through F grade and a note, written while my reasoning is still fresh. I can also write free-form notes that aren't tied to a transaction at all, tagging players with @player. Those turned out to be useful for decisions that didn't show up as a roster move, like trade offers that were rejected or that I cancelled:

Grade: C. I now think this would have been a decent trade. I would have gotten a bump in QB, but would have dropped slightly in TE.

Proposed a trade... I cancelled the trade after 24 hours. I thought I was giving up too much.

The app never grades a move for me. That's deliberate. My grade is my judgment, and the PVS and points deltas are the objective record. The interesting part is comparing the two, and that's where the AI comes in.

The AI Retrospective

The Manager Retrospective is an on-demand report that reads my whole Decision Log (every move's then vs. now numbers, every grade, every note) and writes back four things: how accurate my grades were, my best and worst moves, patterns in my behavior, and a couple of concrete takeaways.

After Part 2, you can probably guess how the prompt is built. The model is handed the data and told not to go beyond it:

If a section's data is thin (few or no grades written, or no points-based
hindsight available on some moves), say so plainly rather than guessing,
do not invent a grade or an outcome that isn't in the data.

It's also told to weigh estimated "then" values more loosely, and to say up front if the log was too long to fit and it's only reading the most recent entries.

Here's some of what it told me:

Alvin Kamara Drop (9/23): At the time, Kamara's drop was supported by a B grade, however, his points jumped from 2.2 to 24.7, illustrating this decision may have been too hasty.

Caleb Douglas Drop (9/23): Graded B, this move turned questionable with Douglas' PVS rocketing from 14.6 to 38.6. The lost long-term potential seems to have been underrated initially.

Focus on Receiver Stability: Establishing a coherent WR strategy could prevent subsequent hesitations and missed opportunities caused by frequent trading and waiver activity...

What surprised me most wasn't that it found mistakes. It was how honest and accurate it was about things I already knew, deep down, and hadn't admitted to myself.

The Pattern I Didn't Want to See

Here's what the retrospective was pointing at. Early in the season, I made a lot of moves based on PVS alone. The number said player X was worth more than player Y, so I swapped them. Then the next week the numbers shifted, and I swapped again. I ended up making more roster moves than anyone else in the league.

Look at how long some of those players actually stayed on my roster:

My note on dropping Kamara literally says "Alvin Kamara was a mistake picking him up." I graded the pickup a D and the drop a B. Then he scored 22.5 more points for somebody else. The mistake wasn't picking him up. It was not giving the decision any time to pay off.

And that's the trap I built for myself in Part 2 without realizing it. PVS is a market number. It moves every time the ranking sites update. If you treat every small shift as a signal to act, you end up chasing the market week to week, and in fantasy football (just like in real markets) that mostly means selling low and buying high.

The retrospective isn't perfect, to be clear. Its write-ups are a bit wordy, and one of its "patterns" read my Mahomes trade offers as chasing immediate production, when I was really a rebuilder trying to get younger at quarterback. But on the big picture, it was right, and it was right about something I didn't want to hear.

What Changed

The biggest change is how I use the app's numbers. I make far fewer moves based on PVS alone. Instead, the numbers are where my research starts: a player with a rising PVS or a strong points trend becomes someone I go read about, not someone I immediately pick up.

You can see the difference in my notes. Early in the season, my reasoning looked like this:

I've been watching Diggs and he looks promising.

A few weeks later:

Brian Robinson Jr has a higher PVS than Kalif Raymond, and is averaging 3 pts higher per week.

That's not a brilliant insight, but it's a decision with actual reasons attached, which means future me can check whether those reasons held up.

I believe it's also made my pickups better. Some of my best moves have been the ones where I looked past the number. Tyler Shough as a backup quarterback has added 74 points since I picked him up. Emanuel Wilson has added 35. The defense and kicker moves above were made on matchups and production, not PVS at all.

So, Is It Luck?

After almost 20 years of saying fantasy football is mostly luck, here's where I've landed.

Luck is real. Injuries, bye weeks, and that one guy who scores three touchdowns against you in the playoffs are all out of anyone's control. But the managers who win most often in my league do it by making better decisions, more consistently, over a long time. I'm not there yet. It's still a learning curve, and my Decision Log will happily show you every bump along the way.

The difference now is that I have a feedback loop. I make a decision, I write down why, and a few weeks later the app tells me (with data, and with an AI that isn't allowed to flatter me) whether I was right.

Wrapping Up the Series

Over these three posts, the Fantasy Football Assistant went from "let the AI pick my team" to something much more useful:

1. Part 1: Testing the "it's all luck" theory, and why an AI with no real data gave me a great-sounding draft and a last-place draft grade.
2. Part 2: Doing the math in code, then grounding the AI in real news so it can explain the numbers without making things up.
3. Part 3 (this post): Recording every decision, grading myself, and letting the data (and a well-constrained AI) show me my blind spots.

If there's one lesson that carries across all three, it's this: AI is most useful when it's working from real data and held to rules, and least useful when you ask it for an opinion. That's true for fantasy football, and I suspect it's true for a lot of the decisions we're all starting to hand to AI.

Now if you'll excuse me, I have a waiver claim to not make.