New proposal: `remember`

News
1

You might want to pull a title, date, or other status off the page and have the AI use it later.

With our proposed remember command, now you can ask TestDriver to remember specific things about what it can see on the screen for use later.

This proposed syntax will allow the AI to set OUTPUT variables for use in later steps. See the video and YAML below for an example.

version: 5.6.7
steps:
  - prompt: search for the current track in spotify
    commands:
      - command: focus-application
        name: spotify
      - command: remember
        output: track
        description: the track now playing in spotify
      - command: hover-text
        text: What do you want to play?
        description: the spotify search bar
        action: click
      - command: type
        text: ${OUTPUT.track}
2

I think this works since remember isn’t an action so much of it is as a dynamic reference. One potential limitation is that this assumes string, but is there a need for something like:

- command: remember
  output: items
  description: Everything in my grocery list

(I think this is why text extraction often has some concept of schema to help the LLM identify structure, casting, and validation)

If you’re interested in prior art, Reference | Cucumber uses <variable> which works well for handling N values and a data table of known values (e.g. a list of users to run this flow against)

1 Like
3

Yes, I think supporting different types is a great idea! Though, then we also need a function to loop through them :slight_smile: