Since a new wave of Generative AI arrived with the release of technologies like Stable Diffusion and ChatGPT, we have continuously explored how to think about them and use them. This page documents some of our experiments.
- TurtleChat: Exploring kids x code in the age of GenAI
- Idea Factory: How can we make AI work when we're not working?
- Autocomplete: Spatial interface for divergent thinking
- SlackGPT (My AI Team): GPT for team augmentation and automation
- Adventure Writer: Assisted writing interface
- Creative Personal Assistant: Speculative prototype for AI personal assistant
When Stable Diffusion and ChatGPT launched, the technologies seemed indistinguishable from magic. No one knew how they technologies worked or how to talk about them, so we used existing metaphors and analogies to try and make sense of them.
When Stable Diffusion was generating an image or ChatGPT would generate text, we would say that they were "dreaming". When they gave us a result that was unexpected or just plain wrong, we would say that they were "hallucinating".
Our goal is to explore new ways of using these technologies, finding new language, new mental models, for tools and products that extend human ability to think and create.
TurtleChat is a work-in-progress chat application designed for kids to draw images based on TurtleArt.
In this working group where all of us have worked with kids, code, and creativity, we are curious about exploring the relationship between children and code in the age of generative AI.
The application uses OpenAI's Assistant API to take in a user prompt, use the TurtleArt reference documentation to generate TurtleArt code, and renders a canvas with the image.
The project is on-going and done in collaboration with Paula Bonta and Brian Silverman from the Playful Invention Company
Prototype: https://turtle-chat.vercel.app/
Example of a conversation with TurtleChat (2x speed)
Idea Factory is a speculative tool that generates images for projects when we are not working.
Every day, after human working hours, the factory spins up and generates images for each active project, based on its latest prompt.
The factory is open from 6pm-midnight, Copenhagen time. This site rebuilds to show the latest factory output at 7am.
Read a longer write-up around the thought and process: idea factory blog post
Prototype: https://ideafactory.superultra.dk/
Example of browsing generated images in an idea factory project
example of 3D autocomplete in action
3D Autocomplete is a quick sketch of an interface that helps come up with multiple suggestions for whatever you're typing, while you remain in control of the typing. Powered by GPT-4.
The app is rendered in 3D, so you can pan/rotate/zoom around.
Try it out at:
https://3dautocomplete.vercel.app
This tweet about a Slack-based AI team inspired the project
First iteration used Zapier for automation
Next iterations used a custom Node.JS server
Each team member has their own persona, defined by a system prompt
Morten Just posted a tweet that set our brains on fire: he had created a Slack bot with different personas based on which slack channel he uses, e.g. #ai-copywriter for copy writing, #ai-web-developer, #ai-lawyer, etc.
We had never built Slack bots before, but thought this was too interesting to pass up.
Through a bunch of fast iterations, we developed SlackGPT, a Slack bot that can act as multiple team members with individual personas.
A longer writeup of the process is available here
Adventure Writer experiments with interactions around assisted writing using GPT.
When ChatGPT arrived, a core use case was to use it to do writing. But we wanted to explore how to use ChatGPT as a supplement and maintain control over the writing process.
The interaction design is really simple: You start writing a story, and when you feel stuck, you press TAB to generate some words. As new words appear, you can use the provided slider to change the amount of words you want to remain.
Creative Personal Assistant was a speculative prototype made to explore the potential of Stable Diffusion, the open source text-to-image AI model that was released August 2022.
With the release of Stable Diffusion, it became clear that these years new wave of generative AI/ML technologies were going to change the technology product landscape and how we work.
To understand technology we need to work with it hands-on. For this project, as usual, I try to combine multiple motivations into a single project:
- Gain hands-on experience with Stable Diffusion
- Use prototyping to speculate how this technology could integrate into daily life
- Explore the concept of AI personal assistants by John Carmack
In an episode of Lex Friedman's podcast featuring John Carmack, he talked about the idea that we will have a collection of AI's as personal assistants.
So we decided to explore a Creative Personal Assistant using Stable Diffusion. Basically the idea is to build a service you can email with a prompt, and receive a reply with the output from stable diffusion. An effortless, informal conversation with a personal assistant. The flow would look something like this:
- Send a prompt to personalassistant@domain.com
- Wait a few minutes
- Receive a reply
We created an e-mail address for the purpose and registered the IMAP/SMTP connection and authentication info.
Then we built a Node.js server that monitors the inbox of e-mail address. When it registers a new e-mail, it extracts the first line from the message body and defines that as the prompt. A text-to-image stable diffusion python-script is then spawned from the server, and when it detects a new output image, the image is e-mailed back to the sender.
- Server receives e-mail. Registers sender and message as prompt
- Server spawns the Stable Diffusion txt2img script
- Server registers image file output from txt2img
- Server sends e-mail back to sender containing image file
10 hours after thinking of the idea, I had a working proof-of-concept prototype. Incredible what's possible with technology nowadays.