What’s really interesting here is that Jason has applied background in using LLMs in anger to create production systems. There’s repeated back and forth about RAG vs. Long Context, vs. Fine Tuning. Which is a big taking point in the AI space right now. Around 12:05 Jason gets into the trade offs between latency and compute. This theme pops up over and over. In short, Jason feels RAG is here to stay, and he makes a compelling case for that.

1. suck data into a data frame 1. do some transforms on that data 1. maybe join to other data 1. get fancy and apply a function to a column in that dataframe 1. dump the result into a database 1. create some graphs or tables from a summary

In a data science or data engineering workflow like that, the only real objects with methods and such are the data frames. And those aren’t my objects. They are Pandas objects. I just pass them around. My experience is that if I try to make my own objects for my type workflows, I just end up with code that’s harder to read and less performant. Neither of whcih interest me. So I just don’t create my own objects in Python.

So when Pydantic was recommended to me I was like, “doesn’t feel like a tool that fits my hand.”

However in this video, Jason Liu lays out a compelling case why Pydantic is the magic glue that allows LLMs to fit inside of software workflows and not randomly break things (much). He lays out a workflow where LLMs return data into Pydantic validator and that helps ensure that the results end up properly parsed and formatted.

Jason created a light weight library called instructor that wraps these ideas so they are easy to use:

Instructor is used in concert with Open.ai function calling to ensure correctly formatted values are going in and out of the model. Seems like a cool approach.

This is worth watching if you’re interested in calling LLMs with code.