answer the question
· Nik
- the bitter lesson is that general methods which scale with computation eventually beat systems built from human knowledge and hand-written rules. today is even more surprising: the closest thing we have produced to intelligence is a shitload of function approximation with trillions of parmeters, scaled until it begins approximating intelligence. truthfully, if you stop there you will just get a sort of holloway-road version of intelligence. or wherever you live that is not-quite-yet-gentrified and rife with the victims of the missteps of your local government, mumbling to itself, walking strangely down the high street. if you take this intelligence and pay a bunch of kenyans to talk to it you will get gpt-3 aka chatgpt that started this whole thing. today's hotness is essentially getting gpt-3 to answer questions hundreds of Times then heuristically distill the outputs, teasing intelligence out of a giant bag of parameters. the teasing out increases in complexity as you switch between "effort" modes on your model of choice. when life gives you lemons eh!
- an even biter-er lesson (i found) is reality has a surprising amount of detail
- i began thinking about a biter-er lesson when i witnessed my boss have his weekly bout of ai psychosis - he has convinced himself and a few others of the fundamental futility of saas. we are a saas bro. sigh
- the idea is simple: who in their right mind would pay for a server when $50 of sota llm tokens can produce software performing any task in a matter of hours?
- what moat does our product, a data analytics platform, have to offer when anyone can buy a $50 pound bash god that can pull data out of any S sources, join and transform it M ways and present it in D differnt ways?
- human mind is a fragile thing.
- ok fine. lets give the bash god a job: answer one question using company data.
- take some user intent, turn it into sql, run it, fetch the results and present them in a useful way. answer the question. ATQ. just so we're on the same page. the expectation is real simple: i ask a question, you give me answer. if i stop wanting the answer after i said i wanted stop working on it. do not leak my database credentials, do not blow up if the answer is large or keep spending my money when i have gone home.
- this is not even a service this is like one function.
- lets implement it
the bash god spits out 20 pleasant-looking lines of python + 6kloc of HTML. i run demmm. ten rows come back, the dashboard is beautiful.
"ok sickkk we got the atq! babyyy!"
"cool, can i see it?"
"yeah, fine, ill send it over."
"it doesn't work."
"what do you mean it doesn't work? it worked here, greg, i just showed you it working."
"it says python not found. what's a python? do i download one?"
"...no. don't touch anything. get someone to help you."
"darren says i also need fourteen packages and the warehouse driver."
"who the fuck is darren?"
"he's helping me set it up."
"fine. thank you darren."
"and can you send it every morning?"
"yes greg, sure, every morning."
"it didn't arrive."
"my laptop was closed, greg, the laptop has to be open to send things."
"so i sent the code to everyone."
"you sent the...w-w-wait, everyone who?"
"they've all improved it."
"all 23 of them changed it? no, greg, theirs isn't right, mine is right. why are they querying the warehouse as me?"
"finance is on the phone."
"heeeeello? yeah whats up you old codger! no i don't know why i have $3000 in warehouse spend."
fine. apparently my one function is a service now.
do one atq
greg asks: "show me revenue by customer this year." the bash god writes some sql. let us be extremely charitable and assume the sql is perfect. we are not blaming the model yet.
the first line needs a connection. on my laptop this was
connect(), with my warehouse credentials sitting nearby and nobody else around to ask what they unlock. on the server there are 23 gregs. which datasource does this question belong to? which credentials open it? is this greg allowed to use them? the generated code cannot carry the answers around in its pockets, so it needs a border it can ask for data across.fine. the border chooses the datasource, applies greg's permissions and opens a connection. unfortunately opening a warehouse connection is blocking work. if our asynchronous server does it directly, every other greg gets to sit quietly while this greg connects.
fine. move it to a worker thread. except warehouse connections are stateful, some drivers expect them to stay on one thread and almost none will stop you sharing one between requests in ways that work beautifully until they suddenly don't. fine. one operation leases one connection. we have invented connection ownership while trying to run one line.
-
this is where the enterprise drivel begins. the border now owns credentials, permissions, warm connections, datasource-specific bs, auditing, metering and eventually caches. these are mostly names for details you discover after greg sends your function to 23 people. ignore them and ATQ becomes a toil sink, a forgotten-ass prototype or the thing that gets you sacked for diverting resources to a project you were not equipped to complete because you hold a history degree from oxford and imagine being able to build anything just by thinking about it.
oh, look the query finished!
now bring the answer back. it might be ten rows or a hundred million; we cannot know beforehand. the pleasant-looking implementation fetches everything, builds a dataframe, turns that into one enormous json document and only then sends it over http. for a while the server may hold the warehouse result, the dataframe and the encoded response at the same time. repeat for every daily report arriving at 9am UTC and brickollilli your server.
fine. stream arrow. yayyy we solved memory.
oh nooo. the warehouse driver is lazy. sql only runs when we request the first batch of the stream, but dumb llm wrote the sql so its kinda wrong and does not compile. but the server already sent
200 OK, before the warehouse noticed a syntax error. we cannot unsend the 200, so an ordinary query error has become a broken stream.fine. pull the first batch before committing the response. now errors raised while submitting the query can still be errors. anything that fails after the first batch is necessarily a broken stream because the response has already begun.
the query works this time! cool. greg changes "this year" to "this quarter" and closes the tab. the socket closes, the python code stops reading and snowflake happily continues calculating the answer nobody wants while recording charges against this month's bill.
fine. cancel the query. except closing an http request does not kill a worker thread and python has no "stop whatever that thread is doing". the cancellation has to travel as a fact: the dead request tells the operation, the operation tells the driver and the driver asks the warehouse to interrupt. every participant has to agree that work without a consumer should stop.
oh nooo. some drivers cancel by making their connection unusable. return that connection to the warm pool and the next greg receives a corpse connector and cant get any data back. fine. interrupted connections get thrown away and healthy ones go back in the pool.
except cancellation can arrive before the query starts, while it waits for a connection, while the warehouse compiles it, between arrow batches or one nanosecond after it finished. every handoff has to check whether the operation is still wanted, while cleanup itself must finish even though the operation has been cancelled.
arrow has a final little present. the stream tells the reader when it is complete. the reader can see that marker and close the response before the producer has finished releasing its warehouse connection. the server may then mistake successful completion for abandonment and interrupt a healthy connection. fine. finish the warehouse work and release its connection before handing greg the final marker.
oh and we were caching the batches while streaming them. cancellation cannot cache half an answer. write to a temporary cache entry, publish it only after the stream finishes and delete it if anything above goes wrong.
we have now implemented "if i stop wanting the answer, stop working on it."
and if the llm simply misunderstood greg, congratulations: do the whole thing OVER AGAIN BAHAHAH.
that was one iteration of one question, assuming the sql was right, the warehouse was reachable and greg meant what he said by "revenue".
the end?
ATQ contains dozens of other innocent-sounding expectations with the same problem. do not worry! i am not writing 69 parts of this.
instead we can turn to the dynamics (ooooooo). getting a point-in-time ATQ right is a composition problem. Anecdotally an llm cannot one shot this today unless you put this blog in the prompt (you're welcome). Throughout this example i generously assumed the llm can actually produce correct sql for the given organisation. Tune in later to find out what T really stands for.