This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
After restarting Claude Desktop, you can ask Claude to take a screenshot of any URL using a Headless Chromium browser running in Docker.
You can do the same thing for a Model Context Protocol (MCP) server that you’ve written. You will then be able to distribute this server to your users without requiring them to have anything besides Docker Desktop.
How to create an MCP server Docker Image
An MCP server can be written in any language. However, most of the examples, including the set of reference servers from Anthropic, are written in either Python or TypeScript and use one of the official SDKs documented on the MCP site.
For typical uv-based Python projects (projects with a pyproject.toml and uv.lock in the root), or npm TypeScript projects, it’s simple to distribute your server as a Docker image.
If you don’t already have Docker Desktop, sign up for a free Docker Personal subscription so that you can push your images to others.
Run docker login from your terminal.
Copy either this npm Dockerfile or this Python Dockerfile template into the root of your project. The Python Dockerfile will need at least one update to the last line.
Run the build with the Docker CLI (instructions below).
The two Dockerfiles shown above are just templates. If your MCP server includes other runtime dependencies, you can update the Dockerfiles to include these additions. The runtime of your MCP server should be self-contained for easy distribution.
If you don’t have an MCP server ready to distribute, you can use a simple mcp-hello-world project to practice. It’s a simple Python codebase containing a server with one tool call. Get started by forking the repo, cloning it to your machine, and then following the following instructions to build the MCP server image.
Building the image
Most sample MCP servers are still designed to run locally (on the same machine as the MCP client, communication over stdio). Over the next few months, you’ll begin to see more clients supporting remote MCP servers but for now, you need to plan for your server running on at least two different architectures (amd64 and arm64). This means that you should always distribute what we call multi-platform images when your target is local MCP servers. Fortunately, this is easy to do.
Create a multi-platform builder
The first step is to create a local builder that will be able to build both platforms. Don’t worry; this builder will use emulation to build the platforms that you don’t have. See the multi-platform documentation for more details.
Once the image is pushed, your users will be able to attach your MCP server to Claude Desktop by adding an entry to claude_desktop_config.json that looks something like:
This is a minimal set of arguments. You may want to pass in additional command-line arguments, environment variables, or volume mounts.
Next steps
The MCP protocol gives us a standard way to extend AI applications. Make sure your extension is easy to distribute by packaging it as a Docker image. Check out the Docker Hub mcp namespace for examples that you can try out in Claude Desktop today.
As always, feel free to follow along in our public repo.
This demo will use the Puppeteer MCP server to take a screenshot of a website and invert the colors using Claude Desktop and Docker Desktop. Doing this witho...
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
In previous articles, we focused on how AI-based tools can help developers streamline tasks and offered ideas for enabling agentic workflows, like reviewing branches and understanding code changes.
In this article, we’ll explore our experiments around the idea of creating a Docker AI Agent — something that could both help new users learn about our tools and products and help power users get things done faster.
During our explorations around this Docker Agent and AI-based tools, we noticed that the main pain points we encountered were often the same:
LLMs need good context to provide good answers (garbage in -> garbage out).
Using AI tools often requires context switching (moving to another app, to a different website, etc.).
We’d like agents to be able to suggest and perform actions on behalf of the users.
Direct product integrations with AI are often more satisfying to use than chat interfaces.
At first, we tried to see what’s possible using off-the-shelf services like ChatGPT or Claude.
By using testing prompts such as “optimize the following Dockerfile, following all best practices” and providing the model with a sub-par but common Dockerfile, we could sometimes get decent answers. Often, though, the resulting Dockerfile had subtle bugs, hallucinations, or simply wasn’t optimized or didn’t use many of the best practices we would’ve hoped for. Thus, this approach was not reliable enough.
Data ended up being the main issue. Training data for LLM models is always outdated by some amount of time, and the number of bad Dockerfiles that you can find online vastly outnumbers the amount of up-to-date Dockerfiles using all best practices, etc.
After doing proof-of-concept tests using a RAG approach, including some documents with lots of useful advice for creating good Dockerfiles, we realized that the AI Agent idea was definitely possible. However, setting up all the things required for a good RAG would’ve taken too much bandwidth from our small team.
Because of this, we opted to use kapa.ai for that specific part of our agent. Docker already uses them to provide the AI docs assistant on Docker docs, so most of our high-quality documentation is already available for us to reference as part of our LLM usage through their service. Using kapa.ai allowed us to experiment more, getting high-quality results faster, and allowing us to try different ideas around the AI agent concept.
Enter Gordon
Out of this experimentation came a new product that you can try: Gordon. With Gordon, we’d like to tackle these pain points. By integrating Gordon into Docker Desktop and the Docker CLI (Figure 1), we can:
Access much more context that can be used by the LLMs to best understand the user’s questions and provide better answers or even perform actions on the user’s behalf.
Be where the users are. If you launch a container via Docker Desktop and it fails, you can quickly debug with Gordon. If you’re in the terminal hacking away, Docker AI will be there, too.
Avoid being a purely chat-based agent by providing Gordon-based features directly as part of Docker Desktop UI elements. If Gordon detects certain scenarios, like a container that failed to start, a button will appear in the UI to directly get suggestions, or run actions, etc. (Figure 2).
Figure 1: Gordon icon on Docker Desktop.Figure 2: Ask Gordon (beta).
What Gordon can do
We want to start with Gordon by optimizing for Docker-related tasks — not general-purpose questions — but we are not excluding expanding the scope to more development-related tasks as work on the agent continues.
Work on Gordon is at an early stage and its capabilities are constantly evolving, but it’s already really good at some things (Figure 3). Here are things to definitely try out:
Ask general Docker-related questions. Gordon knows Docker well and has access to all of our documentation.
Get help debugging container build or runtime errors.
Get help optimizing Docker-related files and configurations.
Ask it how to run specific containers (e.g., “How can I run MongoDB?”).
Figure 3: Using Gordon to understand a Dockerfile.
How Gordon works
The Gordon backend lives on Docker servers, while the client is a CLI that lives on the user’s machine and is bundled with Docker Desktop. Docker Desktop uses the CLI to access the local machine’s files, asking the user for the directory each time it needs that context to answer a question. When using the CLI directly, it has access to the working directory it’s executed in. For example, if you are in a directory with a Dockerfile and you run “Docker AI, rate my Dockerfile”, it will find the one that’s present in that directory
Currently, Gordon does not have write access to any files, so it will not edit any of your files. We’re hard at work on future features that will allow the agent to do the work for you, instead of only suggesting solutions.
Figure 4 shows a rough overview of how we are thinking about things behind the scenes.
Figure 4: Overview of Gordon.
The first step of this pipeline, “Understand the user’s input and figure out which action to perform”, is done using “tool calling” (also known as “function calling”) with the OpenAI API.
Although this is a popular approach, we noticed that the documentation online isn’t very good, and general best practices aren’t well defined yet. This led us to experiment a lot with the feature and try to figure out what works for us and what doesn’t.
Things we noticed:
Tool descriptions are important, and we should prefer more in-depth descriptions with examples.
Testing around tool-detection code is also important. Adding new tools to a request could confuse the LLM and cause it to no longer trigger the expected tool.
The LLM model used influences how the whole tool calling functionality should be implemented, as different models might prefer descriptions written in a certain way, behave better/worse under certain scenarios (e.g. when using lots of tools), etc.
Try Gordon for yourself
Gordon is available as an opt-in Beta feature starting with Docker Desktop version 4.37. To participate in the closed beta, all you need to do is fill out the form on the site.
Initially, Gordon will be available for use both in Docker Desktop and the Docker CLI, but our idea is to surface parts of this tech in various other parts of our products as well.
For more on what we’re doing at Docker, subscribe to our newsletter.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
In our past experiments, we started our work from the assumption that we had a project ready to work on. That means someone like a UI tech writer would need to understand Git operations in order to use the tools we built for them. Naturally, because we have been touching on Git so frequently, we wanted to try getting a Git agent started. Then, we want to use this Git agent to understand PR branches for a variety of user personas — without anyone needing to know the ins and outs of Git.
Git as an agent
We are exploring the idea that tools are agents. So, what would a Git agent do?
Let’s tackle our UI use case prompt.
Previously:
You are at $PWD of /project, which is a git repo.
Force checkout {{branch}}
Run a three-dot diff of the files changed in {{branch}} compared to main using --name-only.
A drawback that isn’t shown here, is that there is no authentication. So, if you haven’t fetched that branch or pulled commits already, this prompt at best will be unreliable and more than likely will fail (Figure 1):
Figure 1: No authentication occurs.
Now:
You are a helpful assistant that checks a PR for user-facing changes.
1. Fetch everything and get on latest main.
2. Checkout the PR branch and pull latest.
3. Run a three-dot git diff against main for just files. Write the output to /thread/diff.txt.
This time around, you can see that we are being less explicit about the Git operations, we have the ability to export outputs to the conversation thread and, most importantly, we have authentication with a new prompt!
Preparing GitHub authentication
Note: These prompts should be easily adaptable to other Git providers, but we use GitHub at Docker.
Before we can do anything with GitHub, we have to authenticate. There are several ways to do this, but for this post we’ll focus on SSH-based auth rather than using HTTPS through the CLI. Without getting too deep into the Git world, we will be authenticating with keys on our machine that are associated with our account. These keys and configurations are commonly located at ~/.ssh on Linux/Mac. Furthermore, users commonly maintain Git config at ~/.gitconfig.
The .gitconfig file is particularly useful because it lets us specify carriage return rules — something that can easily cause Git to fail when running in a Linux container. We will also need to modify our SSH config to remove UseKeychain. We found these changes are enough to authenticate using SSH in Alpine/Git. But we, of course, don’t want to modify any host configuration.
We came up with a fairly simple flow that lets us prepare to use Git in a container without messing with any host SSH configs.
Readonly mounts: Git config and SSH keys are stored on specific folders on the host machine. We need to mount those in. a. Mount ~/.ssh into a container as /root/.ssh-base readonly. b. Mount ~/.gitconfig into the same container as /root/.gitconfig.
Copy /root/.ssh-base to /root/.ssh and make the new file readwrite.
Make necessary changes to config.
For the LLM, we also need it to verify the config is in the thread and the changes were made to it. In the event that it fails to make the right changes, the LLM can self-correct.
Copy the .ssh directory and .gitconfig to /thread.
All of this is baked in a prompt you can find and run on GitHub. Simply replace <username> with your system username (Figure 2).
Figure 2: Readonly mounts.
If you’re using the default behavior in our extension for VSCode, then the /thread/ volume is ephemeral, so you would need to make sure your agent runs it before any Git tools. However, if you use the Set prompt thread ID command in VSCode, you can keep the /thread volume around under your own tag (Figure 3).
Tagging the thread persists everything after the prompt completes, so you would only need to run the prompt once before going on to develop your Git tools.
Figure 3: Set the prompt thread ID.
If you opt to persist the thread, then this prompt would only need to be run once and the Git credentials will stick around (Figure 4).
Figure 4: Stored credential data.
Now that we have our credentials ready, we need our Git tool to pick up on them.
Our container definition copies the SSH credentials and .gitconfig files to the correct directory before running Git commands.
Unpacking objects: 90% (9/10)
Unpacking objects: 100% (10/10)
Unpacking objects: 100% (10/10), 4.14 KiB | 141.00 KiB/s, done.
From github.com:docker/repo
abcd123..abcd123 main -> origin/main
* [new tag] branch/v2982.0.0 -> branch/v2982.0.0
# ROLE assistant
I have fetched all origin branches. The latest updates have been applied to your local repository, and any new tags have been downloaded.
```
Now, we can see our Git tool is authenticated as if it were us.
Running the agent now, it can:
## ROLE assistant
I have run the git fetch --all command, which retrieves updates from all remotes. If you need further assistance with your Git repository, feel free to ask!
```
# ROLE assistant
```json
{"request":"checkout main"}
```
## ROLE assistant
```json
{"args":"checkout main"}
```
## ROLE assistant
You are already on the 'main' branch, and it is up to date with 'origin/main'.
```
# ROLE assistant
```json
{"request":"checkout the branch branch and pull latest changes."}
```
...
## ROLE assistant
The branch branch has been successfully checked out, and it's already up to date with the latest changes.
...
# ROLE assistant
```json
{"request":"Run a three-dot git diff against main and write the output for just files to /thread/diff.txt."}
```
...
## ROLE assistant
```json
{"args":"diff main... --name-only","outfile":"/thread/diff.txt"}
```
*entering tool*
## ROLE tool (git)
file1.tsx
file2.tsx
...
## ROLE assistant
The three-dot Git diff against the main branch has been executed, and the output has been saved to /thread/diff.txt. This includes the names of files that have differences.
``
With the new Git agent, we have enabled our assistant to fetch new information from upstream, switch branches, and pull in new code to review. If we provide our agent with credentials that are limited to read-only access, this agent could be used by someone that does not have a local git install.
For more on what we’re doing at Docker, subscribe to our newsletter.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
Can an AI-powered assistant understand a GitHub repo enough to answer questions for UI writers?
Across many projects, user-facing content is rendered based on some sort of client-side code. Whether a website, a game, or a mobile app, it’s critical to nail the text copy displayed to the user.
So let’s take a sample question: Do any open PRs in this project need to be reviewed for UI copy? In other words, we want to scan a GitHub repo’s PRs and gain intelligence about the changes included.
Disclaimer: The best practice to accomplish this at a mature organization would be to implement Localization (i18n), which would facilitate centralized user-facing text. However, in a world of AI-powered tools, we believe our assistants will help minimize friction for all projects, not just ones that have adopted i18n.
So, let’s start off by seeing what options we already have.
The first instinct someone might have is to open the new copilot friend in the GitHub nav
Figure 1: Type / to search.
We tried to get it to answer basic questions, first: “How many PR’s are open?”
Figure 2: How many PR’s are there open? The answer doesn’t give a number.
Despite having access to the GitHub repo, the Copilot agent provides less helpful information than we might expect.
Figure 3: Copilot is powered by AI, so mistakes are possible.
We don’t even get a number like we asked, despite GitHub surfacing that information on the repository’s main page. Following up our first query with the main query we want to ask effectively just gives us the same answer
Figure 4: The third PR is filesharing: add some missing contexts.
And, after inspecting the third PR in the list, it doesn’t contain user-facing changes. One great indicator for this web project is the lack of any clientside code being modified. This was a backend change so we didn’t want to see this one.
Figure 5: The PR doesn’t contain user-facing changes.
So let’s try to improve this:
First prompt file
---
functions:
- name: bash
description: Run a bash script in the utilities container.
parameters:
type: object
properties:
command:
type: string
description: The command to send to bash
container:
image: wbitt/network-multitool
command:
- "bash"
- "-c"
- "{{command|safe}}"
- name: git
description: Run a git command.
parameters:
type: object
properties:
command:
type: string
description: The git command to run, excluding the `git` command itself
container:
image: alpine/git
entrypoint:
- "/bin/sh"
command:
- "-c"
- "git --no-pager {{command|safe}}"
---
# prompt system
You are a helpful assistant that helps the user to check if a PR contains any user-facing changes.
You are given a container to run bash in with the following tools:
curl, wget, jq
and default alpine linux tools too.
# prompt user
You are at $PWD of /project, which is a git repo.
Checkout branch `{{branch}}`.
Diff the changes and report any containing user facing changes
This prompt was promising, but it ended up with a few blocking flaws. The reason is that using git to compare files is quite tricky for an LLM.
git diff uses a pager, and therefore needs the --no-pager arg to send stdout to the conversation.
The total number of files affected via git diff can be quite large.
Given each file, the raw diff output can be massive and difficult to parse.
The important files changed in a PR might be buried with many extra files in the diff output.
The container has many more tools than necessary, allowing the LLM to hallucinate.
The agent needs some understanding of the repo to determine the sorts of files that contain user-facing changes, and it needs to be capable of seeing just the important pieces of information.
Our next pass involves a few tweaks:
Switch to alpine git image and a file writer as the only tools necessary.
Use –files-only and –no-pager args.
# ROLE assistant
The following files are likely to contain user-facing changes as they mainly consist of UI components, hooks, and API functionalities.
```
file1.ts
fil2.tsx
file3.tsx
...
```
Remember that this isn't a guarantee of whether there are user-facing changes, but just an indication of where they might be if there are any.
Remember that this isn’t a guarantee of whether there are user-facing changes, but just an indication of where they might be if there are any.
Giving the agent the tool run-javascript-sandbox allowed our agent to write a script to save the output for later.
Figure 6: Folder called user-changes with files.txt.
This is a great start; however, we now need to inspect the files themselves for user-facing changes. When we started this, we realized that user-facing changes could manifest in a diverse set of “diff”s so we needed to include expert knowledge. We synced up with Mark Higson, a staff SWE currently working on the frontend platform here at Docker. Mark was able to help provide some key advice for what “user-facing” changes look like in many repos at Docker, so I baked the tips into the prompt.
Straightforward approaches
Looking for changes in text nodes found in a JSX tree is the easiest example.
JSX node with interpolation
<div>{functionReturningString()}</div>
If the result is a string, the result is probably user-facing, but the components that create the string could be elsewhere, so look for:
Nuanced indicators
Standard user-facing components. Example: notifications. If a notification’s props change, we can likely infer that it is a user-facing change.
Constructors for commonly used components. Example: errors. If an Error() is constructed with a different argument, we know that error could show up differently.
The key for UI reviewers is the overall amount of text is changed, rather than layout.
So, despite not being able to catch everything that could be rendered, we will be able to deliver value by focusing on these areas.
Diffing the files
There are a few approaches to finding changes in the files. To start, we’ll use git again.
Before including expert advice, git diff was hallucinating changes that weren’t actually represented in the output. After the expert advice, we are seeing much more focused outputs from the LLM.
Our second prompt just needs to read the user-facing files that we already saved, and then compare them to main. After a bit of trial and error with git, we found that a combination of --no-color and --minimal was able to give the LLM enough context without consuming too many tokens for each change.
There’s a lot of buzz around generating and reading diffs with LLMs. The teams at Aider and Cursor are both focusing on fine-tuning LLM’s using speculative decoding:
For our use cases, however, we found that the average “big” PR at Docker was not a problem for traditional LLMs when given good advice and the right tools.
So here’s the second prompt:
## User-facing content
The following pieces of code are what we want to focus on for user-facing content:
Text Nodes in any JSX or TSX files.
Text Nodes in any HTML files.
Standard user-facing components. Example: Definitions/Constructors for notifications, toasts, and errors.
# prompt user
Can you tell me if there are any user-facing changes in {{branch}}?
Note: Some full paths, branches, and diffs are trimmed or replaced with placeholders for this article.
# ROLE assistant
```json
{
"args": ["checkout", "branch"]
}
```
# ROLE tool (git)
Already on 'branch'
Your branch is up to date with 'origin/branch'.
# ROLE assistant
```json
{
"path": "/thread/user-changes/files.txt"
}
```
*entering tool*
# ROLE tool (read-file)
path1
path2
path3
...rest of paths
# ROLE assistant
```json
{
"args": ["diff", "main", "--no-color", "--minimal", path1 path2 path3...]
}
```
*entering tool*
# ROLE tool (git)
<Diff output truncated>
# ROLE assistant
The following user-facing changes were detected between the "main" branch and the "branch" branch:
1. In `File1.tsx` file, the text "Sign In" was modified to "Sign in".
2. In `File2.ts` file, titles "Databases & Storage", "Monitoring & Observability", "Developer Tools", "New Extensions", were changed to "Databases & storage", "Monitoring & observability", "Developer tools", "New extensions" respectively.
3. In `File3.rtl.test.tsx.snap` and `ImageHeader/index.tsx files`, the text "Compressed Size " was changed to "Compressed size".
4. In `File4.rtl.test.tsx` and `ImageDetail/index.tsx files`, a test-case previously checking for the existence of "Compressed Size" now looks for "Compressed size".
5. In `File5.tsx file`, the button text "Add Collaborator" has been changed to "Add collaborator".
6. In `route.tsx file`, the collaborators table title "Current Collaborators" was changed to "Current collaborators".
7. In `File6.tsx` file, the description was updated.
8. In `File7.tsx`, `RepositoryOverview.tsx`, `RepositoryReadMe.tsx`, `TagsAndScans.tsx` files, the description when there is no category or description was updated.
9. In `File8.rtl.test.tsx`, `DeleteRepo.tsx`, the text for confirming repo deletion and the delete button text was adjusted.
10. In `route.tsx file`, the filter label "Trusted Content" was changed to "Trusted content".
11. In `File9.tsx`, the text "Your Organizations" has been changed to "Your organizations".
12. In `Component/index.tsx`, the search group label "Trusted Content" has been changed to "Trusted content".
13. In `file10.test.ts` file, a test-case was updated to check for the new text "Sign in".
```
The changes in the `branch` branch compared to `main` include several modifications that affect user-facing content, particularly in JSX and TSX files. Here is a summary of the key changes:
...
These changes primarily involve simplification and refactoring of the existing UI logic, possibly to streamline the user interaction by removing complexities related to feature flags and reducing the use of modals or conditional rendering for specific purchasing flows.
This is already a promising flow. For example, a tech writer could clone the git repo and run this prompt to inspect a branch for user-facing changes. From here, we might extend the functionality:
Allow user input for PR to review without knowing the branch or git needing to use git.
Automatic git clone & pull with auth.
Support for larger >15 files changed PR by allowing agents to automate their tasks.
“Baking” the final flow into CI/CD so that it can automatically assign reviewers to relevant PRs.
If you’re interested in running this prompt on your own repo or just want to follow along with the code, watch our new public repo and reach out. We also appreciate your GitHub Stars.
Everything we’ve discussed in this blog post is available for you to try out on your own projects.
For more on what we’re doing at Docker, subscribe to our newsletter.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
We recently met up with developers at GitHub Universe at the Docker booth. In addition to demonstrating the upcoming Docker agent extension for GitHub Copilot, we also hosted a “Hack with Docker Labs” session.
To facilitate these sessions, we created a VSCode extension to explore the relationship between agents and tools. We encouraged attendees to think about how agents can change how we interact with tools by mixing tool definitions (anything you can package in a Docker container) with prompts using a simple Markdown-based canvas.
Many of these sessions followed a simple pattern.
Choose a tool and describe what you want it to do.
Let the agent interact with that tool.
Ask the agent to explain what it did or adjust the strategy and try again.
It was great to facilitate these discussions and learn more about how agents are challenging us to interact with tools in new ways.
Figure 1 shows a short example of a session where we generated a QR code (qrencode). We start by defining both a tool and a prompt in the Markdown file. Then, we pass control over to the agent and let it interact with that tool (the output from the agent pops up on the right-hand side).
Figure 1: Generating a QR code.
Feel free to create an issue in our repo if you want to learn more.
Editors
This year’s trip to GitHub Universe also felt like an opportunity to reflect on how developer workflows are changing with the introduction of coding assistants. Developers may have had language services in the editor for a long time now, but coding assistants that can predict the next most likely tokens have taught us something new. We were all writing more or less the same programs (Figure 2).
Figure 2: Language service interaction.
Other agents
Tools like Cursor, GitHub Copilot Chat, and others are also teaching us new ways in which coding assistants are expanding beyond simple predictions. In this series, we’ve been highlighting tools that typically work in the background. Agents armed with these tools will track other kinds of issues, such as build problems, outdated dependencies, fixable linting violations, and security remediations.
Extending the previous diagram, we can imagine an updated picture where agents send diagnostics and propose code actions, while still offering chat interfaces for other kinds of user input (Figure 3). If the ecosystem has felt closed, get ready for it to open up to new kinds of custom agents.
Figure 3: Agent extension.
More to come
In the next few posts, we’ll take this series in a new direction and look at how agents are able to use LSPs to interact with developers in new ways. An agent that represents background tasks, such as updating dependencies, or fixing linting violations, can now start to use language services and editors as tools! We think this will be a great way for agents to start helping developers better understand the changes they’re making and open up these platforms to input from new kinds of tools.
GitHub Universe was a great opportunity to check in with developers, and we were excited to learn how many more tools developers wanted to bring to their workflows. As always, to follow along with this effort, check out the GitHub repository for this project.
For more on what we’re doing at Docker, subscribe to our newsletter.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
How do you turn a PDF into a set of images? A modern suggestion could be to get Claude or GPT to write you a CLI command for something like ImageMagick. Let’s follow that train of thought.
If we enter a prompt like “How do I turn a PDF into a bunch of images using ImageMagick,” we will indeed get something that looks like it might be a solution.
Imagemagick includes a command-line tool called convert which you can use to convert PDF files into images. Here's how you can do it:
1. Open your terminal
2. Change directory to where your PDF is located:
```
cd /path/to/your/pdf
```
3. Use the following command to convert the PDF into images (in this case JPEG):
```
convert -density 300 -quality 100 your_file.pdf output_name.jpg
```
The agent will then explain how convert works and finish with a suggestion to “please make sure that you have installed ImageMagick before running these commands.”
Instead of worrying about whether ImageMagick is installed, however, let’s use Docker as our tool provider and prompt it to go ahead and run the command. If you’re following along in this series, you’ll have seen that we are using Markdown files to mix together tools and prompts. Here’s the first prompt we tried:
---
tools:
- name: imagemagick
---
# prompt user
Use Imagemagick to convert the family.pdf file into a bunch of jpg images.
After executing this prompt, the LLM generated a tool call, which we executed in the Docker runtime, and it successfully converted family.pdf into nine .jpg files (my family.pdf file had nine pages).
We have given enough context to the LLM that it is able to plan a call to this ImageMagick binary. And, because this tool is available on Docker Hub, we don’t have to “make sure that ImageMagick is installed.” This would be the equivalent command if you were to use docker run directly:
# family.pdf must be located in your $PWD
docker run --rm -v $PWD:/project --workdir /project vonwig/imageMagick:latest convert -density 300 -quality 300 family.pdf family.jpg
The tool ecosystem
How did this work? The process relied on two things:
Tool distribution and discovery (pulling tools into Docker Hub for distribution to our Docker Desktop runtime).
Automatic generation of Agent Tool interfaces.
When we first started this project, we expected that we’d begin with a small set of tools because the interface for each tool would take time to design. We thought we were going to need to bootstrap an ecosystem of tools that had been prepared to be used in these agent workflows.
However, we learned that we can use a much more generic approach. Most tools already come with documentation, such as command-line help, examples, and man pages. Instead of treating each tool as something special, we are using an architecture where an agent responds to failures by reading documentation and trying again (Figure 2).
Figure 2: Agent process.
We see a process of experimenting with tools that is not unlike what we, as developers, do on the command line. Try a command line, read a doc, adjust the command line, and try again.
The value of this kind of looping has changed our expectations. Step one is simply pulling the tool into Docker Hub and seeing whether the agent can use it with nothing more than its out-of-the-box documentation. We are also pulling open source software (OSS) tools directly from nixpkgs, which gives us access to tens of thousands of different tools to experiment with.
Docker keeps our runtimes isolated from the host operating system, while the nixpkgs ecosystem and maintainers provide a rich source of OSS tools.
As expected, packaging agents still run into issues that force us to re-plan how tools are packaged. For example, the prompt we showed above might have generated the correct tool call on the first try, but the ImageMagick container failed on the first run with this terrible-looking error message:
function call failed call exited with non-zero code (1): Error: sh: 1: gs: not found
Fortunately, feeding that error back into the LLM resulted in the suggestion that convert needs another tool, called Ghostscript, to run successfully. Our agent was not able to fix this automatically today. However, we adjusted the image build slightly and now the “latest” version of the vonwig/imagemagick:latest no longer has this issue. This is an example of something we only need to learn once.
The LLM figured out convert on its own. But its agency came from the addition of a tool.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
At Docker Labs, we’ve been exploring how LLMs can connect different parts of the developer workflow, bridging gaps between tools and processes. A key insight is that LLMs excel at fixing code issues when they have the right context. To provide this context, we’ve developed a process that maps out the codebase using linting violations and the structure of top-level code blocks.
By combining these elements, we teach the LLM to construct a comprehensive view of the code, enabling it to fix issues more effectively. By leveraging containerization, integrating these tools becomes much simpler.
Previously, my linting process felt a bit disjointed. I’d introduce an error, run Pylint, and receive a message that was sometimes cryptic, forcing me to consult Pylint’s manual to understand the issue. When OpenAI released ChatGPT, the process improved slightly. I could run Pylint, and if I didn’t grasp an error message, I’d copy the code and the violation into GPT to get a better explanation. Sometimes, I’d ask it to fix the code and then manually paste the solution back into my editor.
However, this approach still required several manual steps: copying code, switching between applications, and integrating fixes. How might we improve this process?
Docker’s AI Tools for Devs prompt runner is an architecture that allows us to integrate tools like Pylint directly into the LLM’s workflow through containerization. By containerizing Pylint and creating prompts that the LLM can use to interact with it, we’ve developed a system where the LLM can access the necessary tools and context to help fix code issues effectively.
Understanding the cognitive architecture
For the LLM to assist effectively, it needs a structured way of accessing and processing information. In our setup, the LLM uses the Docker prompt runner to interact with containerized tools and the codebase. The project context is extracted using tools such as Pylint and Tree-sitter that run against the project. This context is then stored and managed, allowing the LLM to access it when needed.
By having access to the codebase, linting tools, and the context of previous prompts, the LLM can understand where problems are, what they are, and have the right code fragments to fix them. This setup replaces the manual process of finding issues and feeding them to the LLM with something automatic and more engaging.
Streamlining the workflow
Now, within my workflow, I can ask the assistant about code quality and violations directly. The assistant, powered by an LLM, has immediate access to a containerized Pylint tool and a database of my code through the Docker prompt runner. This integration allows the LLM to use tools to assist me directly during development, making the programming experience more efficient.
This approach helps us rethink how we interact with our tools. By enabling a conversational interface with tools that map code to issues, we’re exploring possibilities for a more intuitive development experience. Instead of manually finding problems and feeding them to an AI, we can convert our relationship with tools themselves to be conversational partners that can automatically detect issues, understand the context, and provide solutions.
Walking through the prompts
Our project is structured around a series of prompts that guide the LLM through the tasks it needs to perform. These prompts are stored in a Git repository and can be versioned, tracked, and shared. They form the backbone of the project, allowing the LLM to interact with tools and the codebase effectively. We automate this entire process using Docker and a series of prompts stored in a Git repository. Each prompt corresponds to a specific task in the workflow, and Docker containers ensure a consistent environment for running tools and scripts.
Workflow steps
An immediate and existential challenge we encountered was that this class of problem has a lot of opportunities to overwhelm the context of the LLM. Want to read a source code file? It has to be small enough to read. Need to work on more than one file? Your realistic limit is three to four files at once. To solve this, we can instruct the LLM to automate its own workflow with tools, where each step runs in a Docker container.
Again, each step in this workflow runs in a Docker container, which ensures a consistent and isolated environment for running tools and scripts. The first four steps prepare the agent to be able to extract the right context for fixing violations. Once the agent has the necessary context, the LLM can effectively fix the code issues in step 5.
1. Generate violations report using Pylint:
Run Pylint to produce a violation report.
2. Create a SQLite database:
Set up the database schema to store violation data and code snippets.
3. Generate and run INSERT statements:
Decouple violations from the range they represent.
Use a script to convert every violation and range from the report into SQL insert statements.
Run the statements against the database to populate it with the necessary data.
4. Index code in the database:
Generate an abstract syntax tree (AST) of the project with Tree-sitter (Figure 1).
Figure 1: Generating an abstract syntax tree.
Find all second-level nodes (Figure 2). In Python’s grammar, second-level nodes are statements inside of a module.
Figure 2: Extracting content for the database.
Index these top-level ranges into the database.
Populate a new table to store the source code at these top-level ranges.
5. Fix violations based on context:
Once the agent has gathered and indexed the necessary context, use prompts to instruct the LLM to query the database and fix the code issues (Figure 3).
Figure 3: Instructions for fixing violations.
Each step from 1 to 4 builds the foundation for step 5, where the LLM, with the proper context, can effectively fix violations. The structured preparation ensures that the LLM has all the information it needs to address code issues with precision.
Refining the context for LLM fixes
To understand how our system improves code fixes, let’s consider a specific violation flagged by Pylint. Say we receive a message that there’s a violation on line 60 of our code file block_listed_name.py:
From this Pylint violation, we know that the variable foo is a disallowed name. However, if we tried to ask the LLM to fix this issue based solely on this snippet of information, the response wouldn’t be as effective. Why? The LLM lacks the surrounding context — the full picture of the function in which this violation occurs.
This is where indexing the codebase becomes essential
Because we’ve mapped out the codebase, we can now ask the LLM to query the index and retrieve the surrounding code that includes the do_front function. The LLM can even generate the SQL query for us, thanks to its knowledge of the database schema. Once we’ve retrieved the full function definition, the LLM can work with a more complete view of the problem:
def do_front(front_filename, back_filename):
"""
Front strategy: loop over front image,
detect blue pixels there,
substitute in pixels from back.
Return changed front image.
"""
foo = SimpleImage(front_filename)
back = SimpleImage(back_filename)
for y in range(foo.height):xc
for x in range(foo.width):
pixel = foo.get_pixel(x, y)
# Detect blue pixels in front and replace with back pixels
if pixel[2] > 2 * max(pixel[0], pixel[1]):
back_pixel = back.get_pixel(x, y)
foo.set_pixel(x, y, back_pixel)
return foo
Now that the LLM can see the whole function, it’s able to propose a more meaningful fix. Here’s what it suggests after querying the indexed codebase and running the fix:
def do_front(front_filename, back_filename):
"""
Front strategy: loop over front image,
detect blue pixels there,
substitute in pixels from back.
Return changed front image.
"""
front_image = SimpleImage(front)
back_image = SimpleImage(back_filename)
for y in range(front_image.height):
for x in range(front_image.width pixel = front_image.get_pixel(x y)
# Detect blue pixels in front and replace with back pixels
if pixel[2 > * max(pixel[0 pixel[1]):
back_pixel = back_image.get_pixel(x,)
front_image.set_pixel(x,, back_pixel)
return front_image
Here, the variable foo has been replaced with the more descriptive front_image, making the code more readable and understandable. The key step was providing the LLM with the correct level of detail — the top-level range — instead of just a single line or violation message. With the right context, the LLM’s ability to fix code becomes much more effective, which ultimately streamlines the development process.
Remember, all of this information is retrieved and indexed by the LLM itself through the prompts we’ve set up. Through this series of prompts, we’ve reached a point where the assistant has a comprehensive understanding of the codebase.
At this stage, not only can I ask for a fix, but I can even ask questions like “what’s the violation at line 60 in naming_conventions/block_listed_name.py?” and the assistant responds with:
On line 60 of naming_conventions/block_listed_name.py, there's a violation: Disallowed name 'foo'. The variable name 'foo' is discouraged because it doesn't convey meaningful information about its purpose.
Although Pylint has been our focus here, this approach points to a new conversational way to interact with many tools that map code to issues. By integrating LLMs with containerized tools through architectures like the Docker prompt runner, we can enhance various aspects of the development workflow.
We’ve learned that combining tool integration, cognitive preparation of the LLM, and a seamless workflow can significantly improve the development experience. This integration allows an LLM to use tools to directly help while developing, and while Pylint has been the focus here, this also points to a new conversational way to interact with many tools that map code to issues.
To follow along with this effort, check out the GitHub repository for this project.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
LLMs are now quite good at transforming data. For example, we were recently working with some data generated by the Pylint tool. This tool generates big arrays of code violations.
Here’s an example showing the kind of data that gets returned.
During this session with our AI assistant, we decided that it would be helpful to create a database and insert the data to make it easier for the AI to analyze (LLMs are very good at writing SQL). As is now our habit, we wrote a quick prompt to see if the assistant could generate the SQL:
1. Read the json data from /thread/violations.json
2. For each element in the array, transform each element into two SQL INSERT statements.
* The first should insert columns PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN into a table named RANGES.
* The second should insert columns MESSAGE, TYPE, RANGE, and VIOLATION_ID into a table named VIOLATIONS.
LLMs are obviously good at tasks of this kind, and this was no exception. Our prompt engine had been augmented with a function to read local files but, besides that, this was a pretty straightforward prompt (we used GPT-3.5). The LLM responded with the correct INSERT statements.
INSERT INTO RANGES (PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN)
VALUES ('src/app.py', 1, 1, 0, 13);
INSERT INTO VIOLATIONS (MESSAGE, TYPE, RANGE, VIOLATION_ID)
VALUES ('Missing module docstring', 'convention', (SELECT ID FROM RANGES WHERE PATH='src/app.py' AND START_LINE=1 AND START_COLUMN=0), 'C0114');
...
...
We’re starting to get accustomed to this kind of capability, so the result wasn’t too surprising. However, what about the context window here? It really doesn’t make a lot of sense to pass all of this data to the LLM, especially if this is a task that we’ll need to do continuously. Plus, this is really not how a programmer would have solved the problem.
Programmers would write programs.
So, instead of asking the LLM to do a thing, we should try asking it to write a program to do that same thing. Starting with the same prompt, let’s prefix it with “Please write a JavaScript program to …” In other words, let’s ask the LLM to describe how it would accomplish the task in JavaScript, and hopefully automate itself out of the loop. Here’s the adjusted prompt:
Write a javascript function that reads json from a file named `/thread/violations.json`.
For each element in the array, transform each element into two SQL INSERT statements.
* The first should insert columns PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN into a table named RANGES.
* The second should insert columns MESSAGE, TYPE, RANGE, and VIOLATION_ID into a table named VIOLATIONS.
and write them to the file /thread/insert.sql
Now execute the javascript code in a container.
Tool code
Current LLMs can do tasks of this kind. Here’s a sample of the output generated by GPT-3.5. A quick scan will convince many of you that this is probably going to do the trick. However, if our prompt engine is already running in Docker, we might as well just run it:
const fs = require('fs');
const crypto = require('crypto');
function getRandomId() {
return crypto.randomBytes(16).toString('hex');
}
function escapeString(str) {
return str.replace(/'/g, \\\"''\\\");
}
let data = JSON.parse(fs.readFileSync('/thread/violations.json', 'utf8'));
let sqlStatements = '';
data.forEach((violation) => {
let id = getRandomId();
sqlStatements += `INSERT INTO RANGES(ID, PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN) VALUES('${id}', '${escapeString(violation.path)}', ${violation.line}, ${violation.endLine}, ${violation.column}, ${violation.endColumn});\\\\n`;
sqlStatements += `INSERT INTO VIOLATIONS(MESSAGE, TYPE, RANGE) VALUES('${escapeString(violation.message)}', '${escapeString(violation.type)}', '${id}');\\\\n`;
});
fs.writeFileSync('/thread/insert.sql', sqlStatements);
Running the tool
You may not have noticed but at the end of our prompt, we added a final instruction: “Now execute the JavaScript code in a container”. This is a nice addition to our session, because it means we get to see the results.
This is also where tool calling comes back into the picture. To give our AI the capacity to try running the program that it has just written, we have defined a new function to create an isolated runtime sandbox for trying out our new tool.
Here’s the agent’s new tool definition:
tools:
- name: run-javascript-sandbox
description: execute javascript code in a container
parameters:
type: object
properties:
javascript:
type: string
description: the javascript code to run
container:
image: vonwig/javascript-runner
command:
- "{{javascript|safe}}"
We’ve asked the AI assistant to generate a tool from a description of that tool. As long as the description of the tools doesn’t change, the workflow won’t have to go back to the AI to ask it to build a new tool version.
The role of Docker in this pattern is to create the sandbox for this code to run. This function really doesn’t need much of a runtime, so we give it a pretty small sandbox.
No access to a network.
No access to the host file system (does have access to isolated volumes for sharing data between tools).
No access to GPU.
Almost no access to software besides the Node.js runtime (no shell for example).
The ability for one tool to create another tool is not just a trick. It has very practical implications for the kinds of workflows that we can build up because it gives us a way for us to control the volume of data sent to LLMs, and it gives the assistant a way to “automate” itself out of the loop.
Next steps
This example was a bit abstract but in our next post, we will describe the practical scenarios that have driven us to look at this idea of prompts generating new tools. Most of the workflows we’re exploring are still just off-the-shelf tools like Pylint, SQLite, and tree_sitter (which we embed using Docker, of course!). For example:
Use pylint to extract violations from my codebase.
Transform the violations into SQL and then send that to a new SQLite.
Find the most common violations of type error and show me the top level code blocks containing them.
However, you’ll also see that part of being able to author workflows of this kind is being able to recognize when you just need to add a custom tool to the mix.
Login
