WIP, try telling Claude to filter chunks for relevance

[jrochkind]

Experiment responding to #3278

[jrochkind]

So this change works to exclude irrelevant, but it may work too well, we actually like some "adjacent" answers, especially when nothing better is available?

I feel like this strips out stuff we actually like. We could try to fine tune to strip out less, but I wonder if the list/avoid-interpretation change we made already gets us enough of the way there.

• "Who believes in Aliens?"

  • narrative includes some belief in God, possibly offensively
  • list already eliminates the God answer, although mentions religious beliefs as being related in intro!
  • filtered still mentions religious beliefs in intro. has only ONE item in list, eliminating folks who talked about SETI and possibilty of encoutnering ETs in space exploration -- which were close enough to be interesting! Includes ONLY one guy who does NOT bleieve in ET's, omitting those folks who really sound like they do.

• "Is Boyle's Law still taught?

  • narrative includes totally irrelevant stuff from Laidler about teaching chemistry generally
  • list Laidler is already gone, intro is quite good. Isn't able to answer the question, but has some adjacent mentions of boyle's law that seem useful?
  • filtered just tells us that it can't say whether boyle's law is still taught, full stop. boring.

• "Did any interviewees mention Charles Darwin?"

  • narrative "Including social darwinism in this answer, when not what it was about, and then it's summary of what was discussed was misleading"
    • list still includes social darwinism, but I think the summary is not misleading?
    • filtered still includes social darwinism, the LLM just has trouble realizing that's different than charles darwin the person.

• "What universities are most responsible for the boom in biotechnology?"

  • List format includes irreelvant finding "Charles Cooney describes being recruited by multiple early biotech companies (Cetus, Genex, Genentech) in the late 1970s"
  • filtered Does avoid irrelevant Cooney, but strips out at least one pretty good one too
    • strips out "George Rathmann explains that biotech companies clustered on the coasts rather than the Midwest despite excellent academic centers like Wisconsin and Chicago because Midwestern venture capitalists were extremely risk-averse compared to California investors."

• "Which interviewees won a Nobel Prize before the age of 50?"

  • narrative includes one person that is not an interviewee
  • list still includes non-interviewee
  • filtered success, does eliminate the non-interviewee.

• "What political parties are represented by interviewees? Can you give me a statistical analysis of party affiliation for interviewees who mentioned a party?"

  • list includes people's parents, which are not interviewees
  • filtered still has parents, no change

• "Did any of the interviewee's have relationships with each other"

  • includes relationships NOT with interviewees (the tool can't really answer this at present), and the filtering doesn't chagne the answer, it still does.