Radical just-in-time learning

Buoyed by the success of my LLM-assisted refactoring exercise, I set a new and (for me) more ambitious goal. While the refactoring task was a bit challenging, I didn’t have to learn new concepts and syntax. The basic patterns and Go idioms for Steampipe plugins were already familiar, I just had to find the right way to rearrange things.

Now, though, I wanted to tackle another project that I’d been blocked on: a slider widget for Steampipe dashboards. Why? Well, there are million uses for an input control that reports continuously varying values. For example, the Mastodon dashboard currently limits results by means of a picklist with hardcoded values.

input "limit" {
  title = "Max results"
  type = "select"  
  sql = <<EOQ
    with limits(label) as (
        ( '20' ),
        ( '40' ),
        ( '100' ),
        ( '200' ),
        ( '500' )
      label::int as value

It would be nice, instead, to code a widget like this…

input "limit" {
  title = "Max results"
  type = "slider"
  sql = <<EOQ
      20 as min,
      500 as max,
      40 as step,
      80 as default

…that renders something like this:

Why was this an ambitious goal? In this case there were a whole bunch of new concepts and idioms to absorb. A Steampipe dashboard resource touches many different aspects of the Steampipe system. The main components are: Steampipe itself, the base program that launches both Postgres and the dashboard server; the dashboard server that forwards queries to Steampipe and receives query results; and the dashboard app that defines widgets instantiated with query results. Multiple communication channels are in play, including a websocket connection from the dashboard server to the dashboard app, and event-driven message flow within the app.

For aspiring plugin developers, there’s an SDK and documentation about common idioms and patterns. No such tooling or guidance is yet available to the widget developer. Parts of the dashboard system are written in Go code, parts in React-flavored TypeScript. I’d used TypeScript before, but had never touched React, so I not only needed to learn some key React concepts, I also had to grok how they look and feel in TypeScript.

We’ve seen plenty of examples of LLMs writing Fibonacci generators, then writing tests for them, then translating from Python to JavaScript. Far more typical, I think, are tasks that require diving into an unfamiliar code base and learning just enough to make useful progress on some tactical goal.

Can you fly that thing?

Here’s my favorite scene in the Matrix.

Neo: Can you fly that thing?

(looks toward helicopter)

Trinity: Not yet.

(picks up phone)

Tank: Operator.

Trinity: I need a pilot progam for a B-212 helicopter. Hurry.

(eyelids flutter briefly)

Trinity: Let’s go.

(they jump in the chopper and fly off the roof)

That’s what I want, and what we all need, for coding (among many other things). Could LLM assistants help me rapidly learn what I needed to know to add a basic slider to Steampipe’s gallery of widgets? That was my experiment, and here’s my report.

An overly-optimistic first try

I started by feeding examples of existing input widgets to ChatGPT-4 and Sourcegraph Cody, and asking how to modify the patterns for a new slider widet. Predictably that yielded confident but useless results. Although Sourcegraph has now indexed the Steampipe repo, and is presumably making embeddings available to Cody, it wasn’t clear to me whether, or to what extent, that made a difference. No big deal, I wasn’t really expecting this approach to work, but it never hurts to ask!

Next I tried prompting with a José Reyes blog post that contains a detailed explanation of how he made a significant improvement to the chart widget. Would that material supply enough useful context? Not for ChatGPT-4 or Cody, evidently. If there’s a way to craft a prompt that uses José’s explanation to produce a working slider, I’d love to see it.

If you haven’t seen José’s blog, by the way, I commend it to you. Over the past few months he’s shown an astonishing ability to jump into diverse code bases and do useful things with them. His Steampipe series, in six installments so far, covers a wide variety of Steampipe topics in great depth and provides a wealth of instructive examples.

It’s humbling, to be honest. I’ve worked for a while now on the periphery of the Steampipe, building plugins and dashboards, but have barely scratched the surface as a contributor to the core. Although I’m wary of applying the 10x label to a developer, José clearly deserves it. Me, not so much, 1x at best. I’m a fountain of good ideas that I often struggle to implement quickly and well, especially in code bases that span unfamiliar languages, frameworks, and components. If LLM augmentation can get me even to 1.5x or 2x, never mind 10x, that’ll be a huge win for me — and for a whole lot of average developers like me.

A more realistic next try

Programmers aim to decompose tasks into small, manageable chunks. The same discipline applies when working with LLM assistants. For starters, I lowered the goal post. I originally hoped to implement a two-handled slider, like the one I used in this AV editor. But first I needed a basic understanding of how to wire up any kind of new dashboard widget, and for that I wanted the simplest possible slider.

I asked both ChatGPT-4 and Cody, in various ways, for ideas. At one point, Cody proposed <input type="range">. Oh? No kidding? I didn’t know that was a thing, thanks Cody! A native HTML element was just the right starter element. There would still be plenty to learn about Steampipe/React/TypeScript conventions for wiring such a thing into Steampipe’s dashboard system. No need to complicate matters with a library that implements a fancier slider but brings extra conceptual baggage.

As a widget developer, you’re operating in an environment where the results of your widget’s query are magically available in a data property that you can unpack in the widget’s TypeScript code. How is that data made available to the widget? It’s complicated. I spent a good while spelunking to find out how exactly it’s done, I still can’t clearly articulate all the ingredients, or data flows that interconnect them, and I didn’t find a way for the LLMs to usefully aid the investigation. At one point I found myself in Chromium’s devtools, looking at websocket traffic and setting breakpoints to isolate event sources and handlers. I can envision LLMs enlisting debuggers, just as ChatGPT can now consult WolframAlpha via a plugin, but we’re not there yet. Manual debugging (where available!) will remain a critical skill for exploring unfamiliar code.

Tracing that data flow wasn’t strictly necessary, though. For my purposes, it was OK to just accept that the data property magically holds the results of the query. Now I needed to work out how to use that data to initialize the widget, and then wire up a handler to set a new value when the slider’s handle moves. As a React/TypeScript newbie, though, even that was nontrivial. Last time I tried I got lost in the abstractions and came up empty-handed. This time, I hoped my pair of rubber ducks could help me break through.

Exploring examples

I modeled the slider on another widget type, select, which builds a picklist and reports the selected item. It wasn’t an ideal starting point because there’s a family of select widgets, interrelated in ways that require more complex coding than the slider would. Neither of the LLMs managed to distill the essence of a slider from those examples, but in the back-and-forth dialog with them I began to put the pieces together.

It was straightford to put a slider onto the canvas. Then it took me a while to work out how to wire it into the React machinery. At one point, as multiple useEffect hooks came into play, I found myself wondering about why their dependency arrays were sometimes empty. Both LLMs gave the same answer: an empty array means the effect only runs once after the initial render.

I’m an example-driven learner, though, so I wanted to see which of the useEffect hooks in the code base work that way. ChatGPT-4 can’t see the whole repo, so it doesn’t know. Cody, in VSCode, does see the repo, and is further enlightened by the embeddings in Sourcegraph’s indexed copy of the repo. Cody found some of the examples, but not all. Then it occurred to me to try Sourcegraph’s structural search. Here’s the query that found all the examples.

context:global repo:^github\.com/turbot/steampipe$ useEffect(..., []) 

It’s interesting to see that, while Cody is joined at the hip to the Sourcegraph web app — and indeed even runs in a sidebar there, as well as in VSCode — it doesn’t yet seem to tap into structural search. Perhaps that will become the first external tool that plugs into Cody?

Controlled vs uncontrolled components

At one point, as I worked out how to persist the slider’s selected value, I encountered what I now realize is a common React error.

Warning: A component is changing a controlled input to be uncontrolled.

Me: Huh?

Cody: This warning is occurring because you are setting a value prop on the element, but also listening for onChange events. This is conflicting – either the input should be controlled (by setting the value prop) or uncontrolled (by listening for onChange events). To fix this, you have a few options:

  • Make the input controlled by setting value and handling changes with an onChange handler that calls setValue().

  • Make the input uncontrolled by removing the value prop and keeping the onChange handler.

  • Use a ref to access the DOM input and read its value when needed: const inputRef = useRef<HTMLInputElement>(null);

Eventually I worked out how to do it both ways. Then I asked ChatGPT-4 and Cody to compare them, evaluate tradeoffs, and weigh in on which approach to take. One favored uncontrolled’s simplicity, the other liked controlled’s flexiblity. Ultimately that’s for the maintainer to decide, if this turns into a pull request. For now I’m sticking with controlled, with a better sense of why than I could otherwise have easily gained.

A catalog of concepts

Beyond the controlled/uncontrolled distinction, here (courtesy of ChatGPT-4) are some of the concepts and idioms that this exercise introduced me to:

  • Functional Component: SliderInput is defined as a functional component. In TypeScript, the props that this component accepts are explicitly typed as InputProps.
  • Props and TypeScript Interfaces: In TypeScript, interfaces are used to ensure the props have the correct type. The data and name props are passed to the SliderInput component, and their types are defined by the InputProps interface.
  • Destructuring Props: The props are destructured in the function parameter, allowing the properties to be accessed directly as variables (data and name) instead of as properties on a props object.
  • useState Hook with TypeScript: The useState hook is used to add state to functional components. In TypeScript, you can provide a type argument to this hook to specify the type of the state variable. In this case, value is defined as a state variable with the type number.
  • useEffect Hook: The useEffect hook lets you perform side effects in function components. In this case, it’s used for two purposes:

    • To update the value state when the stateValue prop changes.
    • To dispatch an action to set the initial value of the slider when the defaultValue changes.
  • Custom Hooks with TypeScript: Custom hooks are a mechanism to reuse stateful logic between components. The useDashboard hook is used to access the dispatch function and selectedDashboardInputs state from a dashboard context. These returned values also have their own types defined.
  • Event Handling with TypeScript: Event handlers are used to handle user interactions with the component. In this case, the sliderChange function is triggered when the slider value changes. The type of the event object e is inferred from the context, but it could also be explicitly defined with React.ChangeEvent<HTMLInputElement> for stricter type checking.
  • JSX and TypeScript (TSX): JSX is a syntax extension for JavaScript that is used to describe what the UI should look like. When using TypeScript, this syntax is referred to as TSX. TypeScript provides additional type safety in TSX code.

The output of this exercise was a few dozen lines of code involving all that conceptual stuff. But the result of the exercise feels bigger. As a developer of Steampipe dashboards, there are a few other affordances I’d like to have. Yes, it’s open source, so I can contribute, but I’m hard-pressed to justify climbing that learning curve. The LLM assistants ease that curve; I’m a better learner with them than without them; more feels possible than before. Doug Engelbart dreamed of augmenting human intellect. I wish he’d lived to see this moment.

Posted in .

3 thoughts on “Radical just-in-time learning

Leave a Reply