Developers are increasingly expected to integrate AI capabilities into their applications but they also face many challenges. Namely, the steep learning curve, coupled with an overwhelming array of tools and frameworks, makes this process too tedious. Docker aims to bridge this gap with the Docker AI Catalog, a curated experience designed to simplify AI development and empower both developers and publishers.
Why Docker for AI?
Docker and container technology has been a key technology used by developers at the forefront of AI applications for the past few years. Now, Docker is doubling down on that effort with our AI Catalog. Developers using Docker’s suite of products are often responsible for building, deploying, and managing complex applications — and, now, they must also navigate generative AI (GenAI) technologies, such as large language models (LLMs), vector databases, and GPU support.
For developers, the AI Catalog simplifies the process of integrating AI into applications by providing trusted and ready-to-use content supported by comprehensive documentation. This approach removes the hassle of evaluating numerous tools and configurations, allowing developers to focus on building innovative AI applications.
Key benefits for development teams
The Docker AI Catalog is tailored to help users overcome common hurdles in the evolving AI application development landscape, such as:
Decision overload: The GenAI ecosystem is crowded with new tools and frameworks. The Docker AI Catalog simplifies the decision-making process by offering a curated list of trusted content and container images, so developers don’t have to wade through endless options.
Steep learning curve: With the rise of new technologies like LLMs and retrieval-augmented generation (RAG), the learning curve can be overwhelming. Docker provides an all-in-one resource to help developers quickly get up to speed.
Complex configurations preventing production readiness: Running AI applications often requires specialized hardware configurations, especially with GPUs. Docker’s AI stacks make this process more accessible, ensuring that developers can harness the full power of these resources without extensive setup.
The result? Shorter development cycles, improved productivity, and a more streamlined path to integrating AI into both new and existing applications.
Empowering publishers
For Docker verified publishers, the AI Catalog provides a platform to differentiate themselves in a crowded market. Independent software vendors (ISVs) and open source contributors can promote their content, gain insights into adoption, and improve visibility to a growing community of AI developers.
Key features for publishers include:
Increased discoverability: Publishers can highlight their AI content within a trusted ecosystem used by millions of developers worldwide.
Metrics and insights: Verified publishers gain valuable insights into the performance of their content, helping them optimize strategies and drive engagement.
Unified experience for AI application development
The AI Catalog is more than just a repository of AI tools. It’s a unified ecosystem designed to foster collaboration between developers and publishers, creating a path forward for more innovative approaches to building applications supported by AI capabilities. Developers get easy access to essential AI tools and content, while publishers gain the visibility and feedback they need to thrive in a competitive marketplace.
With Docker’s trusted platform, development teams can build AI applications confidently, knowing they have access to the most relevant and reliable tools available.
The road ahead: What’s next?
Docker will launch the AI Catalog in preview on November 12, 2024, alongside a joint webinar with MongoDB. This initiative will further Docker’s role as a leader in AI application development, ensuring that developers and publishers alike can take full advantage of the opportunities presented by AI tools.
Stay tuned for more updates and prepare to dive into a world of possibilities with the Docker AI Catalog. Whether you’re an AI developer seeking to streamline your workflows or a publisher looking to grow your audience, Docker has the tools and support you need to succeed.
Ready to simplify your AI development process? Explore the AI Catalog and get access to trusted content that will accelerate your development journey. Start building smarter, faster, and more efficiently.
For publishers, now is the perfect time to join the AI Catalog and gain visibility for your content. Become a trusted source in the AI development space and connect with millions of developers looking for the right tools to power their next breakthrough.
This #MagPiMonday, we’re hoping to inspire you to add artificial intelligence to your Raspberry Pi designs with this feature by Phil King, from the latest issue of The MagPi.
With their powerful AI accelerator modules, Raspberry Pi’s Camera Module and AI Kit open up exciting possibilities in computer vision and machine learning. The versatility of the Raspberry Pi platform, combined with AI capabilities, opens up a world of new possibilities for innovative smart projects. From creative experiments to practical applications like smart pill dispensers, makers are harnessing the kit’s potential to push the boundaries of AI. In this feature, we explore some standout projects, and hope they inspire you to embark on your own.
AI computer vision can identify objects within a live camera view. In this project, VEEB’s Martin Spendiff and Vanessa Bradley have used it to detect humans in the frame, so you can tell if your boss is approaching behind you as you sit at your desk!
The project comprises two parts. A Raspberry Pi 5 equipped with a Camera Module and AI Kit handles the image recognition and also acts as a web server. This uses web sockets to send messages wirelessly to the ‘detector’ part — a Raspberry Pi Pico W and a voltmeter whose needle moves to indicate the level of AI certainty for the ID.
Having got their hands on an AI Kit — “a nice intro into computer vision” — it took the pair just three days to create Peeper Pam. “The most challenging bit was that we’d not used sockets — more efficient than the Pico constantly asking Raspberry Pi ‘do you see anything?’,” says Martin. “Raspberry Pi does all the heavy lifting, while Pico just listens for an ‘I’ve seen something’ signal.”
While he notes that you could get Raspberry Pi 5 to serve both functions, the two-part setup means you can place the camera in a different position to monitor a spot you can’t see. Also, by adapting the code from the project’s GitHub repo, there are lots of other uses if you get the AI to deter other objects. “Pigeons in the window box is one that we want to do,” Martin says.
Never one to do things by halves, Jeff Geerling went overboard with Raspberry Pi AI Kit and built a Monster AI Pi PC with a total of eight neural processors. In fact, with 55 TOPS (trillions of operations per second), it’s faster than the latest AMD, Qualcomm, and Apple Silicon processors!
The NPU chips — including the AI Kit’s Hailo-8L — are connected to a large 12× PCIe slot card with a PEX 8619 switch capable of handling 16 PCI Express Gen 2 lanes. The card is then mounted on a Raspberry Pi 5 via a Pineboards uPCIty Lite HAT, which has an additional 12V PSU to supply the extra wattage needed for all those processors.
With a bit of jiggery-pokery with the firmware and drivers on Raspberry Pi, Jeff managed to get it working.
As a proof of concept, Japanese maker Naveen aimed to implement an automated system for identifying and monitoring cars at toll plazas to get an accurate tally of the vehicles entering and exiting.
With the extra processing power provided by a Raspberry AI Kit, the project uses Edge Impulse computer vision to detect and count cars in the view from a Camera Module Wide. “We opted for a wide lens because it can capture a larger area,” he says, “allowing the camera to monitor multiple lanes simultaneously.” He also needed to train and test a YOLOv5 machine learning model. All the details can be found on the project page via the link above, which could prove useful for learning how to train custom ML models for your own AI project.
Wearing a safety helmet on a building site is essential and could save your life. This computer vision project uses Raspberry Pi AI Kit with the advanced YOLOv8 machine learning model to quickly and accurately identify objects within the camera view, running at an impressive inference speed of 30fps.
The project page has a guide showing how to make use of Raspberry Pi AI Kit to achieve efficient AI inferencing for safety helmet detection. This includes details of the software installation and model training process, for which the maker has provided a link to a dataset of 5000 images with bounding box annotations for three classes: helmet, person, and head.
Google’s MediaPipe is an open-source framework developed for building machine learning pipelines, especially useful for working with videos and images.
Having used MediaPipe on other platforms, Mario Bergeron decided to experiment with it on a Raspberry Pi AI Kit. On the project page (linked above) he details the process, including using his Python demo application with options to detect hands/palms, faces, or poses.
Mario’s test results show how much better the AI Kit’s Hailo-8L AI accelerator module performs compared to running reference TensorFlow Lite models on Raspberry Pi 5 alone: up to 5.8 times faster. With three models running for hand and landmarks detection, the frame rate is 26–28fps with one hand detected, and 22–25fps for two.
The MagPi #147 out NOW!
You can grab the new issue right now from Tesco, Sainsbury’s, Asda, WHSmith, and other newsagents, including the Raspberry Pi Store in Cambridge. It’s also available at our online store, which ships around the world. You can also get it via our app on Android or iOS.
You can also subscribe to the print version of The MagPi. Not only do we deliver it globally, but people who sign up to the six- or twelve-month print subscription get a FREE Raspberry Pi Pico W!
This October has been tremendously packed with my duty as Seeed Studio’s Ranger. From OSHWAto Pycon APAC 2024, every event left a great spot on my mind. Here is the recap for my activity this October 2024.
1. October 3rd 2024, OSHWA 24 Hours Membership Drive Show & Tell
This live online session is initiated by OSHWA and runs 24 hours straight. My session is scheduled to run at 6 PM local time. After rushing to be at home before my schedule, I managed to get on time and share my open-source projects on Hackster. After sharing my open-source projects, I introduce the Seeed Studio Co-Create Gadget campaign to the audience.
2. October 17th 2024, Transforming IoT with Edge Intelligence Seminar
This seminar is initiated by the student organization on my campus and has two speakers. The other speaker is one of my ex student whom now work as a cloud engineer. It is the perfect opportunity to introduce Seeed Studio to my campus while also giving the students insight into the latest technology in AI, Machine Learning, IoT, and Edge Computing.
In the seminar, I mainly focus on the next step of IoT which is the inclusivity of Machine Learning in IoT solutions and how reComputer could be a small compact solution to implement it.
3. October 25th 2024, Pycon APAC 2024
This is my highlight of the month because the Python Indonesia community is becoming the host for this year’s annual Python Conference of the Asia Pacific area. In total, there are 80 Speakers from across Asia pacific with hundreds of participants.
At Pycon I managed to schedule a workshop session on 25th October 2024 titled “Building IoT and Camera Project with XIAO ESP32S3 and Micropython”.
In this workshop, I shared various things that the XIAO ESP32S3 can do with Micropython such as basic input and output, camera stream, displaying JSON weather data on the OLED screen, and interacting with the Grove Vision AI V2 sensor.
4. November 2nd 2024, Internet of Things Seminar at Universitas Banten Jaya
A friend from another campus has invited me to this seminar. The location of campus is in Serang, a neighboring city from where I live and it is a 2-hour drive from home.
The material that I shared in this seminar is the same as what I shared at my previous seminar on my campus. The organizer told me that they don’t have much insight regarding IoT on campus, so this seminar could be a huge step for advancing the knowledge on their campus.
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
How do you turn a PDF into a set of images? A modern suggestion could be to get Claude or GPT to write you a CLI command for something like ImageMagick. Let’s follow that train of thought.
If we enter a prompt like “How do I turn a PDF into a bunch of images using ImageMagick,” we will indeed get something that looks like it might be a solution.
Imagemagick includes a command-line tool called convert which you can use to convert PDF files into images. Here's how you can do it:
1. Open your terminal
2. Change directory to where your PDF is located:
```
cd /path/to/your/pdf
```
3. Use the following command to convert the PDF into images (in this case JPEG):
```
convert -density 300 -quality 100 your_file.pdf output_name.jpg
```
The agent will then explain how convert works and finish with a suggestion to “please make sure that you have installed ImageMagick before running these commands.”
Instead of worrying about whether ImageMagick is installed, however, let’s use Docker as our tool provider and prompt it to go ahead and run the command. If you’re following along in this series, you’ll have seen that we are using Markdown files to mix together tools and prompts. Here’s the first prompt we tried:
---
tools:
- name: imagemagick
---
# prompt user
Use Imagemagick to convert the family.pdf file into a bunch of jpg images.
After executing this prompt, the LLM generated a tool call, which we executed in the Docker runtime, and it successfully converted family.pdf into nine .jpg files (my family.pdf file had nine pages).
We have given enough context to the LLM that it is able to plan a call to this ImageMagick binary. And, because this tool is available on Docker Hub, we don’t have to “make sure that ImageMagick is installed.” This would be the equivalent command if you were to use docker run directly:
# family.pdf must be located in your $PWD
docker run --rm -v $PWD:/project --workdir /project vonwig/imageMagick:latest convert -density 300 -quality 300 family.pdf family.jpg
The tool ecosystem
How did this work? The process relied on two things:
Tool distribution and discovery (pulling tools into Docker Hub for distribution to our Docker Desktop runtime).
Automatic generation of Agent Tool interfaces.
When we first started this project, we expected that we’d begin with a small set of tools because the interface for each tool would take time to design. We thought we were going to need to bootstrap an ecosystem of tools that had been prepared to be used in these agent workflows.
However, we learned that we can use a much more generic approach. Most tools already come with documentation, such as command-line help, examples, and man pages. Instead of treating each tool as something special, we are using an architecture where an agent responds to failures by reading documentation and trying again (Figure 2).
Figure 2: Agent process.
We see a process of experimenting with tools that is not unlike what we, as developers, do on the command line. Try a command line, read a doc, adjust the command line, and try again.
The value of this kind of looping has changed our expectations. Step one is simply pulling the tool into Docker Hub and seeing whether the agent can use it with nothing more than its out-of-the-box documentation. We are also pulling open source software (OSS) tools directly from nixpkgs, which gives us access to tens of thousands of different tools to experiment with.
Docker keeps our runtimes isolated from the host operating system, while the nixpkgs ecosystem and maintainers provide a rich source of OSS tools.
As expected, packaging agents still run into issues that force us to re-plan how tools are packaged. For example, the prompt we showed above might have generated the correct tool call on the first try, but the ImageMagick container failed on the first run with this terrible-looking error message:
function call failed call exited with non-zero code (1): Error: sh: 1: gs: not found
Fortunately, feeding that error back into the LLM resulted in the suggestion that convert needs another tool, called Ghostscript, to run successfully. Our agent was not able to fix this automatically today. However, we adjusted the image build slightly and now the “latest” version of the vonwig/imagemagick:latest no longer has this issue. This is an example of something we only need to learn once.
The LLM figured out convert on its own. But its agency came from the addition of a tool.
Hello everyone. This week mostly involved a lot of chasing stuff around (sometimes in vain), so while there was not much headline work, this post might end up a bit longer than usual. Let’s get started without delay.
BeagleConnect Freedom Adventures
I started the week by trying to add IEEE802154 subg socket support in MicroPython for BeagleConnect Freedom. However, I quickly learned that it would not be simple. For some reason, BeagleConnect Freedom would completely lock up after starting.
Initially, I thought it might be a MicroPython issue, so I tried tracking down where the freeze happened. This, however, led to a dead end since, for some reason, the program would not be able to read from the UART console. While doing this, I also remembered a similar issue I encountered while working on BeagleConnect Freedom rc car demo. At the time, I fixed it by just removing unused config items like ADC and PWM from config, but forgot about it after the OOSC conference.
After some experimenting with PWM, ADC, and IEEE802154 subg radio, I figured out that the problem is reproducible in other Zephyr samples like echo_cliet, etc. For some reason, if both PWM pins (MB1 PWM and MB2 PWM) are enabled alongside the subg radio, everything freezes. If one or both of the PWM are disabled, everything works fine. This seems to be an issue with timers but it needs further investigation.
With the MicroPython issue and a bricked BeagleConnect Freedom, I thought it is a good time to setup and learn Ti’s Code Composer Studio.
I use Fedora Sway Atomic as my daily driver, and thus mostly rely on flatpaks or podman containers. However, running Code Composer Studio inside a podman container (created using toolbox) was not a great experience for me. It would randomly stutter (maybe a hardware acceleration problem?) and freeze. Additionally, while udev can make it almost painless to handle device permissions, it can occasionally cause hiccups with flashing. In fact, one of the primary reasons I switched to neovim was that my emacs GUI kept having weird performance problems inside the container.
So, I finally went ahead and installed CCS Theia on my base system. The install procedure is a bit weird since there is no rpm or deb package. Instead, there is an installer which installs everything in $HOME/ti folder. It also creates an uninstall, which seems to work. All in all, while I prefer a flatpack or app image, it wasn’t too bad.
I hit a snag quite early on when I was unable to flash the cc1352p1 on my launchpad. I tried various things and opened a Ti E2E question for the same. However, the solution turned out to be quite weird. I was not saving my workspace since, well, nothing was working anyway, and CCS Theia would open the unsaved workspace. But everything magically worked once I saved my workspace because I was tired of the dialog on exit. Not really sure why.
Once I could flash the launchpad, I tried using the xds110 in launchpad with my BeagleConnect Freedom. I was able to flash a simple blinky on it and even set up breakpoints.
Now, I need to figure out how to use openocd and add instructions in Beagle docs and Zephyr docs for the same.
KUnit Adventures
I have been working on kernel patches that require writing some unit tests. So I was trying to get KUnit to work. However, kunit run kept on failing for some reason, even with the default config. The output was not very clear either. However, after following some debugging instructions, I found out that I could not execute the user mode kernel from inside the podman container. I have created an issue in Fedora toolbox regarding the same.
MicroPython
I have added MicroPython support for BeaglePlay cc1352p7 in my draft PR. It supports IEEE802154 subg sockets and also helped me ensure that MicroPython networking should work fine on BeagleConnect Freedom as well once the timer issue is resolved.
Since BeaglePlay cc1352p7 Zephyr support was merged after the 3.7.0 release, the MicroPython support will continue to live in the draft PR until MicroPython supports a newer Zephyr version.
Zephyr
Zephyr support for BeagleBoard boards continues to improve. We will continue to work to make Beagle one of the best-supported platforms for Zephyr development.
BeagleBone AI-64
Thanks to the work by Andrew Davis, Zephyr support for R5 cores in BeagleBone AI64 was merged this week. Here is the Zephyr page for BeagleBone AI 54. This adds one more board to the growing list of BeagleBoard boards that support Zephyr.
BeagleY-AI
A PR for Zephyr support was opened by Andrew Davis after BBAI-64 support was merged. Anyone interested should feel free to try it out. Hopefully, it can get merged upstream soon.
BeagleBoard Imager Rust Updates
While working on BeagleY-AI, I found a bug in the sha256 handling of the Rust-based imager while translating the old bb-imager config. So, I have created release 0.0.2 for the imager. I probably should implement changelogs before the next release.
Ending Thoughts
This was it for this week. Hopefully, this helps bring transparency regarding where the development efforts are concentrated, and how the community can help. Look forward to next update.
The AI HAT+ features the same best-in-class Hailo AI accelerator technology as our AI Kit, but now with a choice of two performance options: the 13 TOPS (tera-operations per second) model, priced at $70 and featuring the same Hailo-8L accelerator as the AI Kit, and the more powerful 26 TOPS model at $110, equipped with the Hailo-8 accelerator.
Designed to conform to our HAT+ specification, the AI HAT+ automatically switches to PCIe Gen 3.0 mode to maximise the full 26 TOPS of compute power available in the Hailo-8 accelerator.
Unlike the AI Kit, which utilises an M.2 connector, the Hailo accelerator chip is directly integrated onto the main PCB. This change not only simplifies setup but also offers improved thermal dissipation, allowing the AI HAT+ to handle demanding AI workloads more efficiently.
What can you do with the 26 TOPS model over the 13 TOPS model? The same, but more… You can run more sophisticated neural networks in real time, achieving better inference performance. The 26 TOPS model also allows you to run multiple networks simultaneously at high frame rates. For instance, you can perform object detection, pose estimation, and subject segmentation simultaneously on a live camera feed using the 26 TOPS AI HAT+:
Both versions of the AI HAT+ are fully backward compatible with the AI Kit. Our existing Hailo accelerator integration in the camera software stack works in exactly the same way with the AI HAT+. Any neural network model compiled for the Hailo-8L will run smoothly on the Hailo-8; while models specifically built for the Hailo-8 may not work on the Hailo-8L, alternative versions with lower performance are generally available, ensuring flexibility across different use cases.
After an exciting few months of AI product releases, we now offer an extensive range of options for running inferencing workloads on Raspberry Pi. Many such workloads – particularly those that are sparse, quantised, or intermittent – run natively on Raspberry Pi platforms; for more demanding workloads, we aim to be the best possible embedded host for accelerator hardware such as our AI Camera and today’s new Raspberry Pi AI HAT+. We are eager to discover what you make with it.
Today, the Open Source Initiative has released its first official definition of Open Source AI. This is an important milestone. Let me explain why.
Why Open Source matters
In a world where speech depends on software, free speech depends on free software.
The key tenet of open source is is that it puts the user in control. Software is ever growing in complexity and importance in our society. Software is how we do our work, how we communicate, how we pay, how we access information. When software is a black box, subject to subtle (or not so much) manipulation, users are at risk.
Risks of AI
AI brings these risks to an entirely new level. Not only because it makes decisions that are often entirely intransparent, but also because its easy, human like interface often lulls users to trust it far more than deserved.
The big AI firms have done their level best to ensure the attention of the public around risks with AI were aimed at existential, contrived notions akin to Skynet, the AI in the movie series Terminator, scenarios where AI would take over the world. In reality, the risks associated with AI are far more mundane. Surveillance, bias, explosive energy usage and job losses are the concerns we should focus on.
Need for control
And, just like with software, what matters is control. Who controls the AI, who makes decisions about what it can and can’t do, what goes in and what does not. With control, we can address the real risks of AI. Without control, we can simply hope that the billion dollar companies do the right thing.
They haven’t in the past. So we need Open Source AI. AI that that gives users the ability to study and modify the AI models that govern their lives.
Nextcloud Ethical AI Rating
Nextcloud gave this a first shot in March 2023, when we launched our Ethical AI Rating. A simple traffic light would show with green/yellow/orange/red if a given AI model was freely available, if its data was publicly available and if the code needed to run and train it was open source. This way we help users make an informed decision without restricting their choice of models and features.
Users of AI solutions deserve transparency and control, which is why we introduced our Ethical AI rating in early 2023. Now, we see big tech firms trying to hijack the term open source AI. We fully endorse the creation of a clear definition of open source AI by the community to protect users and the market.
Frank Karlitschek
CEO and fouder of Nextcloud
The wider open source community has picked up the gauntlet as well, and after extensive consultation with the community, today the OSI has announced an official definition of Open Source AI. This will help users, from private users, to governments, research institutes, hospitals and businesses, to make decisions what systems they can trust.
So, today, It is a first step in a journey and we are glad to be a part of it. Nextcloud has formally endorsed the definition, even if we think there is room for improvement. We will use it as a basis for our Ethical AI Rating. Our rating is a bit more granular and also more critical in some areas – for example, when it comes to data, we believe it should be always fully available – and thus, for now, we will keep using it, as it fits the use cases of our users more.
We look forward to your input, both on the OSI definition – on the road to a 2.0 – and on our AI rating.
SiFive HiFive Premier P550 is a mini-DTX (203 x 170mm) motherboard powered by a 1.4 GHz ESWIN EIC7700X quad-core RISC-V SiFive P550 SoC with up to 19.95 TOPS of AI performance, and equipped with up to 32GB LPDDR5 memory and a 128GB eMMC flash all soldered on a system-on-module. The motherboard itself features a SATA III connector for data storage, includes an HDMI 2.0 port for 4K video output, a PCIe Gen3 x16 slot (working at x4), two gigabit Ethernet ports, an M.2 Key-E socket to add a WiFi/Bluetooth card, up to five USB interfaces, and more. HiFive Premier P550 specifications: SoC – ESWIN EIC7700X CPU 4x SiFive Performance P550 RV64GC RISC-V cores @ 1.4GHz (up to 1.8GHz when overclocked) with Cortex-A75-class performance 32KB(I) + 32KB(D) L1 Cache 256KB L2 Cache 4MB shared L3 Cache Cache supports ECC (support SECDED) NPU (Not currently supported in software) – Up to 19.95 [...]
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
At Docker Labs, we’ve been exploring how LLMs can connect different parts of the developer workflow, bridging gaps between tools and processes. A key insight is that LLMs excel at fixing code issues when they have the right context. To provide this context, we’ve developed a process that maps out the codebase using linting violations and the structure of top-level code blocks.
By combining these elements, we teach the LLM to construct a comprehensive view of the code, enabling it to fix issues more effectively. By leveraging containerization, integrating these tools becomes much simpler.
Previously, my linting process felt a bit disjointed. I’d introduce an error, run Pylint, and receive a message that was sometimes cryptic, forcing me to consult Pylint’s manual to understand the issue. When OpenAI released ChatGPT, the process improved slightly. I could run Pylint, and if I didn’t grasp an error message, I’d copy the code and the violation into GPT to get a better explanation. Sometimes, I’d ask it to fix the code and then manually paste the solution back into my editor.
However, this approach still required several manual steps: copying code, switching between applications, and integrating fixes. How might we improve this process?
Docker’s AI Tools for Devs prompt runner is an architecture that allows us to integrate tools like Pylint directly into the LLM’s workflow through containerization. By containerizing Pylint and creating prompts that the LLM can use to interact with it, we’ve developed a system where the LLM can access the necessary tools and context to help fix code issues effectively.
Understanding the cognitive architecture
For the LLM to assist effectively, it needs a structured way of accessing and processing information. In our setup, the LLM uses the Docker prompt runner to interact with containerized tools and the codebase. The project context is extracted using tools such as Pylint and Tree-sitter that run against the project. This context is then stored and managed, allowing the LLM to access it when needed.
By having access to the codebase, linting tools, and the context of previous prompts, the LLM can understand where problems are, what they are, and have the right code fragments to fix them. This setup replaces the manual process of finding issues and feeding them to the LLM with something automatic and more engaging.
Streamlining the workflow
Now, within my workflow, I can ask the assistant about code quality and violations directly. The assistant, powered by an LLM, has immediate access to a containerized Pylint tool and a database of my code through the Docker prompt runner. This integration allows the LLM to use tools to assist me directly during development, making the programming experience more efficient.
This approach helps us rethink how we interact with our tools. By enabling a conversational interface with tools that map code to issues, we’re exploring possibilities for a more intuitive development experience. Instead of manually finding problems and feeding them to an AI, we can convert our relationship with tools themselves to be conversational partners that can automatically detect issues, understand the context, and provide solutions.
Walking through the prompts
Our project is structured around a series of prompts that guide the LLM through the tasks it needs to perform. These prompts are stored in a Git repository and can be versioned, tracked, and shared. They form the backbone of the project, allowing the LLM to interact with tools and the codebase effectively. We automate this entire process using Docker and a series of prompts stored in a Git repository. Each prompt corresponds to a specific task in the workflow, and Docker containers ensure a consistent environment for running tools and scripts.
Workflow steps
An immediate and existential challenge we encountered was that this class of problem has a lot of opportunities to overwhelm the context of the LLM. Want to read a source code file? It has to be small enough to read. Need to work on more than one file? Your realistic limit is three to four files at once. To solve this, we can instruct the LLM to automate its own workflow with tools, where each step runs in a Docker container.
Again, each step in this workflow runs in a Docker container, which ensures a consistent and isolated environment for running tools and scripts. The first four steps prepare the agent to be able to extract the right context for fixing violations. Once the agent has the necessary context, the LLM can effectively fix the code issues in step 5.
1. Generate violations report using Pylint:
Run Pylint to produce a violation report.
2. Create a SQLite database:
Set up the database schema to store violation data and code snippets.
3. Generate and run INSERT statements:
Decouple violations from the range they represent.
Use a script to convert every violation and range from the report into SQL insert statements.
Run the statements against the database to populate it with the necessary data.
4. Index code in the database:
Generate an abstract syntax tree (AST) of the project with Tree-sitter (Figure 1).
Figure 1: Generating an abstract syntax tree.
Find all second-level nodes (Figure 2). In Python’s grammar, second-level nodes are statements inside of a module.
Figure 2: Extracting content for the database.
Index these top-level ranges into the database.
Populate a new table to store the source code at these top-level ranges.
5. Fix violations based on context:
Once the agent has gathered and indexed the necessary context, use prompts to instruct the LLM to query the database and fix the code issues (Figure 3).
Figure 3: Instructions for fixing violations.
Each step from 1 to 4 builds the foundation for step 5, where the LLM, with the proper context, can effectively fix violations. The structured preparation ensures that the LLM has all the information it needs to address code issues with precision.
Refining the context for LLM fixes
To understand how our system improves code fixes, let’s consider a specific violation flagged by Pylint. Say we receive a message that there’s a violation on line 60 of our code file block_listed_name.py:
From this Pylint violation, we know that the variable foo is a disallowed name. However, if we tried to ask the LLM to fix this issue based solely on this snippet of information, the response wouldn’t be as effective. Why? The LLM lacks the surrounding context — the full picture of the function in which this violation occurs.
This is where indexing the codebase becomes essential
Because we’ve mapped out the codebase, we can now ask the LLM to query the index and retrieve the surrounding code that includes the do_front function. The LLM can even generate the SQL query for us, thanks to its knowledge of the database schema. Once we’ve retrieved the full function definition, the LLM can work with a more complete view of the problem:
def do_front(front_filename, back_filename):
"""
Front strategy: loop over front image,
detect blue pixels there,
substitute in pixels from back.
Return changed front image.
"""
foo = SimpleImage(front_filename)
back = SimpleImage(back_filename)
for y in range(foo.height):xc
for x in range(foo.width):
pixel = foo.get_pixel(x, y)
# Detect blue pixels in front and replace with back pixels
if pixel[2] > 2 * max(pixel[0], pixel[1]):
back_pixel = back.get_pixel(x, y)
foo.set_pixel(x, y, back_pixel)
return foo
Now that the LLM can see the whole function, it’s able to propose a more meaningful fix. Here’s what it suggests after querying the indexed codebase and running the fix:
def do_front(front_filename, back_filename):
"""
Front strategy: loop over front image,
detect blue pixels there,
substitute in pixels from back.
Return changed front image.
"""
front_image = SimpleImage(front)
back_image = SimpleImage(back_filename)
for y in range(front_image.height):
for x in range(front_image.width pixel = front_image.get_pixel(x y)
# Detect blue pixels in front and replace with back pixels
if pixel[2 > * max(pixel[0 pixel[1]):
back_pixel = back_image.get_pixel(x,)
front_image.set_pixel(x,, back_pixel)
return front_image
Here, the variable foo has been replaced with the more descriptive front_image, making the code more readable and understandable. The key step was providing the LLM with the correct level of detail — the top-level range — instead of just a single line or violation message. With the right context, the LLM’s ability to fix code becomes much more effective, which ultimately streamlines the development process.
Remember, all of this information is retrieved and indexed by the LLM itself through the prompts we’ve set up. Through this series of prompts, we’ve reached a point where the assistant has a comprehensive understanding of the codebase.
At this stage, not only can I ask for a fix, but I can even ask questions like “what’s the violation at line 60 in naming_conventions/block_listed_name.py?” and the assistant responds with:
On line 60 of naming_conventions/block_listed_name.py, there's a violation: Disallowed name 'foo'. The variable name 'foo' is discouraged because it doesn't convey meaningful information about its purpose.
Although Pylint has been our focus here, this approach points to a new conversational way to interact with many tools that map code to issues. By integrating LLMs with containerized tools through architectures like the Docker prompt runner, we can enhance various aspects of the development workflow.
We’ve learned that combining tool integration, cognitive preparation of the LLM, and a seamless workflow can significantly improve the development experience. This integration allows an LLM to use tools to directly help while developing, and while Pylint has been the focus here, this also points to a new conversational way to interact with many tools that map code to issues.
To follow along with this effort, check out the GitHub repository for this project.
We’re excited to introduce #Virgil, an innovative project that combines the power of Arduino technology with a passion for history, creating a groundbreaking interactive experience for museums.
UsingArduino’s versatile and scalable ecosystem, #Virgil operates completely offline, allowing visitors to interact with 3D avatars in a seamless and immersive way. The project brings the past to life, offering dialogue-driven encounters with key historical figures thanks to voice recognition and edge AI – with the option to choose among many different languages.
“#Virgil is meant to celebrate the past and, more importantly, open new avenues for education and inspiration. We want to prove how technology, when guided by ethical values, can amplify and perpetuate our cultural heritage in ways that used to be unimaginable,” comments Enrico Benevenuta, coordinator of the Territori Svelati project and AI expert.
Matteo Olivetti, great-grandson of Olivetti’s founder Camillo, drew inspiration from the iconic Divisumma to design a dedicated hardware setup, Olivox.
Powered by the Portenta X8 and Max Carrier, the device connects via HDMI to any screen, engaging visitors in a rich, interactive experience without the need for smartphones or a stable internet connection. This approach allows the project to adapt easily to different exhibitions and contexts, while offering full control over the visitor experience.
Internationally renowned 3D artist Elvis Morelli was entrusted with creating the first avatar of the project – and it’s no coincidence that Camillo Olivetti was chosen.
The story of Olivetti resonates deeply with Arduino’s own mission of pushing the boundaries of technology, and #Virgil represents a continuation of that legacy by bridging the gap between the past and future through cutting-edge tools.
To find out more about the project and perhaps have a chat with your favorite pioneer of technology and innovation, visit #Virgil’s booth at the upcomingMaker Faire Rome 2024, booth E.09. Don’t forget to stop by Arduino’s booth N.07 to find out more about our products, and let us know what you asked Camillo!
MIT Media Lab researchers Cathy Mengying Fang, Patrick Chwalek, Quincy Kuang, and Pattie Maes have developed WatchThis, a groundbreaking wearable device that enables natural language interactions with real-world objects through simple pointing gestures. Cathy conceived the idea for WatchThis during a one-day hackathon in Shenzhen, organized as part of MIT Media Lab’s “Research at Scale” initiative. Organized by Cedric Honnet and hosted by Southern University of Science and Technology and Seeed Studio, the hackathon provided the perfect setting to prototype this innovative device using components from the Seeed Studio XIAO ESP32S3 suite. By integrating Vision-Language Models (VLMs) with a compact wrist-worn device, WatchThis allows users to ask questions about their surroundings in real-time, making contextual queries as intuitive as pointing and asking.
Credit: Cathy Fang
Hardwares
The WatchThis project utilizes the following hardware components:
WatchThis is designed to seamlessly integrate natural, gesture-based interaction into daily life. The wearable device consists of a watch with a rotating, flip-up camera attached to the back of a display. When the user points at an object of interest, the camera captures the area, and the device processes contextual queries based on the user’s gesture.
The interaction begins when the user flips up the watch body to reveal the camera, which then captures the area where the finger points at. The watch’s display shows a live feed from the camera, allowing precise aiming. When the user touches the screen, the device captures the image and pauses the camera feed. The captured RGB image is then compressed into JPG format and converted to base64, after which an API request is made to query the image.
The device uses these API calls to interact with OpenAI’s GPT-4o model, which accepts both text and image inputs. This allows the user to ask questions such as “What is this?” or “Translate this,” and receive immediate responses. The text response is displayed on the screen, overlaid on the captured image. After the response is shown for 3 seconds, the screen returns to streaming the camera feed, ready for the next command.
The software driving WatchThis is written in Arduino-compatible C++ and runs directly on the device. It is optimized for quick and efficient performance, with an end-to-end response time of around 3 seconds. Instead of relying on voice recognition or text-to-speech—which can be error-prone and resource-intensive—the system uses direct text input for queries. Users can further personalize their interactions by modifying the default query prompt through an accompanying WebApp served on the device, allowing tailored actions such as identifying objects, translating text, or requesting instructions.
Credit: Cathy Fang
Applications
Imagine strolling through a city and pointing at a building to learn its history, or identifying an exotic plant in a botanical garden with a mere gesture.
The device goes beyond simple identification, offering practical applications like real-time translation of, for example, menu items, which is a game-changer for travelers and language learners alike.
The research team has discussed even more exciting potential applications:
A “Remember this” function could serve as a visual reminder system, potentially aiding those who need to take medication regularly.
For urban explorers, a “How do I get there” feature could provide intuitive, spatially-aware navigation by allowing users to point at distant landmarks.
A “Zoom in on that” capability could offer a closer look at far-off objects without disrupting the user’s activities.
Perhaps most intriguingly, a “Turn that off” function could allow users to control smart home devices with a combination of voice commands and gestures, seamlessly integrating with IoT ecosystems.
While some of these features are still in conceptual stages, they paint a picture of a future where our interactions with the world around us are more intuitive, informative, and effortless than ever before.
Credit: Cathy Fang
Build Your Own WatchThis
Interested in building your own WatchThis wearable? Explore the open-source hardware and software components on GitHub and start creating today! Check out their paper below for full details.
Hey community, we’re curating a monthly newsletter centering around the beloved Seeed Studio XIAO. If you want to stay up-to-date with:
Cool Projects from the Community to get inspiration and tutorials Product Updates: firmware update, new product spoiler Wiki Updates: new wikis + wiki contribution News: events, contests, and other community stuff
The ASUS NUC 14 Pro AI is a powerful, compact mini PC featuring the Intel Core Ultra processor (Series 2), which integrates CPU, GPU, and NPU architectures to deliver up to 120 platform TOPS for AI processing. It is designed for consumer, commercial, and edge computing applications. According to the Asus product announcement earlier last […]
Qualcomm Technologies has introduced the new industrial-grade IQ chipset family with the IQ9, IQ8, and IQ6 series offering on-device AI performance of up to 100 TOPS, industrial temperature range, and built-in safety features such as SIL-3 (safety and integrity level). Qualcomm IQ series of chipsets target a range of premium (IQ9), mid-tier (IQ8), and entry-level (IQ6) applications such as industrial and agricultural robots, drones, industrial inspection and automation, edge AI boxes with computer vision capabilities, edge gateway analytics solutions, and more. Qualcomm IQ9 Series – IQ-9100, IQ-9075 Key features and specifications: CPU IQ-9075 – Octa-core Kryo Gen 6 scaling from 1.632 to 2.55 GHz IQ-9100 – Octa-core Kryo Gold Prime @ 2.36 GHz (CNXSoft: The specs are not clear for the CPU part…, both SKUs could be the same for the CPU part) GPU – Adreno 663 GPU Audio DSP (LPASS) 1980 MPPS, 7x TDM/I2S 3x High-Speed I2S for Radio [...]
PALO ALTO, CA– JetBrains, the creators of PyCharm, the popular Python IDE for data science and web development, has formed a new partnership with OpenCV, the world’s largest library for computer vision. As part of the collaboration, JetBrains has joined OpenCV as a Silver Member, making PyCharm the official Python IDE for OpenCV.
Actively developed since June 2000, OpenCV is essential for developers and researchers working in fields like artificial intelligence (AI), machine learning, and robotics, providing powerful, open-source tools that accelerate innovation. JetBrains’ financial contribution as a Silver Member will help sustain OpenCV.org, ensuring that this invaluable resource remains free for both commercial and non-commercial projects. This is especially important as more and more of the tech industry becomes closed off to the open source community.
JetBrains, known for its suite of world-class development tools, including PyCharm, has a long-standing reputation for delivering innovative software solutions. PyCharm, in particular, is a favorite among developers due to its smart code completion, deep code analysis, support for web development frameworks, and interactive Jupyter notebooks. In addition, PyCharm is powered by an AI Assistant and provides superior database support, Python script editing, as well as support for Hugging Face, Databricks, Conda, dbt-Core, and much more. Its slogan, “Focus on code and data. PyCharm will take care of the rest,” reflects the platform’s mission to let developers focus on their core tasks while PyCharm automates routine processes. This is especially beneficial for developers working with OpenCV, as it ensures that AI and data science projects are developed faster, more efficiently, and with fewer errors.
Dr. Satya Mallick, CEO of OpenCV, expressed enthusiasm for the partnership, saying, “High-quality development tools like PyCharm are essential for driving innovation in AI and computer vision. JetBrains’ support as a Silver Member ensures that OpenCV continues to be freely available for developers around the world. PyCharm’s powerful features will undoubtedly enhance productivity and spark the imagination of OpenCV community members everywhere.”
A JetBrains executive commented, “At JetBrains, giving back to the community is a core part of our mission. By partnering with OpenCV, we’re supporting a global ecosystem of developers working in AI and computer vision, ensuring they have the best tools and open-source resources available. Our collaboration with OpenCV reflects our commitment to advancing technology and empowering developers to focus on what matters: creating impactful code.”
JetBrains’ involvement in OpenCV will also be highlighted on OpenCV Live, a popular weekly show which airs Thursday at 9am Pacific. PyCharm will be featured in episodes that showcase how its features can enhance the development process for computer vision and AI applications, beginning with an appearance on November 7th. Registration for the stream is available at http://opencv.live
As an industry leader, JetBrains has long been committed to supporting the open-source community. Trusted by developers worldwide, including those at companies like Google, Microsoft, and Meta, JetBrains provides tools that improve productivity and foster innovation. The company’s decision to become an OpenCV Silver Member reinforces its dedication to the advancement of AI and computer vision, two fields that are rapidly transforming industries around the world.
For organizations interested in joining JetBrains in supporting open-source computer vision and AI, OpenCV offers a variety of membership opportunities. Becoming a member allows companies to contribute directly to the sustainability of OpenCV, ensuring that these powerful tools remain accessible to all.
More information on how to support OpenCV’s mission can be found at opencv.org/membership
About JetBrains JetBrains is a software development company renowned for creating powerful, intelligent tools designed to enhance developer productivity. Founded in 2000, JetBrains offers a wide range of integrated development environments (IDEs) and tools tailored to various programming languages and platforms. Among its flagship products are PyCharm, a leading Python IDE that provides robust features for coding, debugging, and testing, and CLion, an advanced IDE for C and C++ development. JetBrains’ tools are trusted by developers worldwide to streamline workflows, improve code quality, and foster efficient development across multiple programming environments.
About OpenCV OpenCV is the largest and most widely used open-source library for computer vision and artificial intelligence. The library is downloaded over 20 million times per month, and used in an estimated 80% of embedded vision systems. OpenCV code powered Stanley, the first DARPA Grand Challenge winner, and was used in NASA’s 2020 Mars Helicopter project. Operated by the non-profit Open Source Vision Foundation, OpenCV is dedicated to advancing the field through open collaboration and democratizing access to transformative technologies. OpenCV’s official website is https://opencv.org
We are happy to announce the release of Stalwart Mail Serverv0.10.3, which introduces support for AI models —a powerful new feature now available to Enterprise Edition users as well as our GitHub and OpenCollective sponsors. With this feature, Stalwart Mail Server can be integrated with both self-hosted and cloud-based Large Language Models (LLMs), bringing advanced email processing capabilities like never before.
This integration allows you to use AI models for a variety of tasks, including enhanced spam filtering, threat detection, and intelligent email classification. Whether you choose to host your own models with LocalAI or leverage cloud-based services like OpenAI or Anthropic, this release provides the flexibility to incorporate cutting-edge AI into your email infrastructure.
With the introduction of AI model integration, Stalwart Mail Server can now analyze email content more deeply than traditional filters ever could. For instance, in the realm of spam filtering and threat detection, AI models are highly effective at identifying patterns and detecting malicious or unsolicited content. The system works by analyzing both the subject and body of incoming emails through the lens of an LLM, providing more accurate detection and filtering.
In addition to bolstering security, AI integration enhances email classification. By configuring customized prompts, administrators can instruct AI models to categorize emails based on their content, leading to more precise filtering and organization. This is particularly useful for enterprises managing a high volume of messages that span various topics and departments, as AI-driven filters can quickly and intelligently sort messages into categories like marketing, personal correspondence, or work-related discussions.
The flexibility of using either self-hosted or cloud-based AI models means that Stalwart can be tailored to your infrastructure and performance needs. Self-hosting AI models ensures full control over data and privacy, while cloud-based models offer ease of setup and access to highly optimized, continuously updated language models.
One of the most exciting features of this release is the ability for users and administrators to access AI models directly from Sieve scripts. Stalwart extends the Sieve scripting language by introducing the llm_prompt function, which allows users to send prompts and email content to the AI model for advanced processing.
For example, the following Sieve script demonstrates how an AI model can be used to classify emails into specific folders based on the content:
require ["fileinto", "vnd.stalwart.expressions"]; # Base prompt for email classification let "prompt" '''You are an AI assistant tasked with classifying personal emails into specific folders. Your job is to analyze the email's subject and body, then determine the most appropriate folder for filing. Use only the folder names provided in your response. If the category is not clear, respond with "Inbox". Classification Rules: - Family: * File here if the message is signed by a Doe family member * The recipient's name is John Doe - Cycling: * File here if the message is related to cycling * File here if the message mentions the term "MAMIL" - Work: * File here if the message mentions "Dunder Mifflin Paper Company, Inc." or any part of this name * File here if the message is related to paper supplies * Only classify as Work if it seems to be part of an existing sales thread or directly related to the company's operations - Junk Mail: * File here if the message is trying to sell something and is not work-related * Remember that John lives a minimalistic lifestyle and is not interested in purchasing items - Inbox: * Use this if the message doesn't clearly fit into any of the above categories Analyze the following email and respond with only one of these folder names: Family, Cycling, Work, Junk Mail, or Inbox. '''; # Prepare the base Subject and Body let "subject" "thread_name(header.subject)"; let "body" "body.to_text"; # Send the prompt, subject, and body to the AI model let "llm_response" "llm_prompt('gpt-4', prompt + '\n\nSubject: ' + subject + '\n\n' + body, 0.6)"; # Set the folder name if eval "contains(['Family', 'Cycling', 'Work', 'Junk Mail'], llm_response)" { fileinto "llm_response"; }
This example demonstrates how the llm_prompt function can be used to classify emails into different categories such as Family, Cycling, Work, or Junk Mail based on the content. The AI model analyzes the message’s subject and body according to the classification rules defined in the prompt and returns the most appropriate folder name. The email is then automatically filed into the correct folder, making it easier to organize incoming messages based on their content.
With this new feature, Stalwart Mail Server allows for seamless integration with both self-hosted and cloud-based AI models. If you prefer full control over your infrastructure, you can opt to deploy models on your own hardware using solutions like LocalAI. Self-hosting gives you complete ownership over your data and ensures compliance with privacy policies, but it may require significant computational resources, such as GPU acceleration, to maintain high performance.
Alternatively, you can integrate with cloud-based AI providers like OpenAI or Anthropic, which offer access to powerful, pretrained models with minimal setup. Cloud-based models provide cutting-edge language processing capabilities, but you should be aware of potential costs, as these providers typically charge based on the number of tokens processed. Whether you choose self-hosted or cloud-based models, Stalwart gives you the flexibility to tailor the AI integration to your specific needs.
This exciting AI integration feature is exclusively available for Enterprise Edition users as well as GitHub and OpenCollective monthly sponsors. If you want to harness the full potential of AI-powered email processing in Stalwart Mail Server, upgrading to the Enterprise Edition or becoming a sponsor is a great way to access this feature and other advanced capabilities.
The release of Stalwart Mail Server v0.10.3 marks a major milestone in our journey toward building intelligent, highly customizable email management solutions. By combining traditional email filtering with the power of LLMs, Stalwart gives you the tools to take your email infrastructure to the next level, enhancing security, organization, and automation in ways that were previously impossible. We’re excited to see how you’ll use this new feature to optimize your email workflows!
People have been using Raspberry Pi products to build artificialintelligenceprojects for almost as long as we’ve been making them. As we’ve released progressively more powerful devices, the range of applications that we can support natively has increased; but in any generation there will always be some workloads that require an external accelerator, like the Raspberry Pi AI Kit, which we launched in June.
The AI Kit is an awesomely powerful piece of hardware, capable of performing thirteen trillion operations per second. But it is only compatible with Raspberry Pi 5, and requires a separate camera module to capture visual data. We are very excited therefore to announce a new addition to our camera product line: the Raspberry Pi AI Camera.
The AI Camera is built around a Sony IMX500 image sensor with an integrated AI accelerator. It can run a wide variety of popular neural network models, with low power consumption and low latency, leaving the processor in your Raspberry Pi free to perform other tasks.
Key features of the Raspberry Pi AI Camera include:
12 MP Sony IMX500 Intelligent Vision Sensor
Sensor modes: 4056×3040 at 10fps, 2028×1520 at 30fps
1.55 µm × 1.55 µm cell size
78-degree field of view with manually adjustable focus
Integrated RP2040 for neural network and firmware management
The AI Camera can be connected to all Raspberry Pi models, including Raspberry Pi Zero, using our regular camera ribbon cables.
Using Sony’s suite of AI tools, existing neural network models using frameworks such as TensorFlow or PyTorch can be converted to run efficiently on the AI Camera. Alternatively, new models can be designed to take advantage of the AI accelerator’s specific capabilities.
Under the hood
To make use of the integrated AI accelerator, we must first upload a model. On older Raspberry Pi devices this process uses the I2C protocol, while on Raspberry Pi 5 we are able to use a much faster custom two-wire protocol. The camera end of the link is managed by an on-board RP2040 microcontroller; an attached 16MB flash device caches recently used models, allowing us to skip the upload step in many cases.
Once the sensor has started streaming, the IMX500 operates as a standard Bayer image sensor, much like the one on Raspberry Pi Camera Module 3. An integrated Image Signal Processor (ISP) performs basic image processing steps on the sensor frame (principally Bayer-to-RGB conversion and cropping/rescaling), and feeds the processed frame directly into the AI accelerator. Once the neural network model has processed the frame, its output is transferred to the host Raspberry Pi together with the Bayer frame over the CSI-2 camera bus.
Integration with Raspberry Pi libcamera
A key benefit of the AI Camera is its seamless integration with our Raspberry Pi camera software stack. Under the hood, libcamera processes the Bayer frame using our own ISP, just as it would for any sensor.
We also parse the neural network results to generate an output tensor, and synchronise it with the processed Bayer frame. Both of these are returned to the application during libcamera’s request completion step.
The Raspberry Pi camera frameworks — Picamera2 and rpicam-apps, and indeed any libcamera-based application — can retrieve the output tensor, correctly synchronised with the sensor frame. Here’s an example of an object detection neural network model (MobileNet SSD) running under rpicam-apps and performing inference on a 1080p video at 30fps.
This demo uses the postprocessing framework in rpicam-apps to generate object bounding boxes from the output tensor and draw them on the image. This stage takes no more than 300 lines of code to implement. An equivalent application built using Python and Picamera2 requires many fewer lines of code.
Another example below shows a pose estimation neural network model (PoseNet) performing inference on a 1080p video at 30fps.
Although these examples were recorded using a Raspberry Pi 4, they run with the same inferencing performance on a Raspberry Pi Zero!
Together with Sony, we have released a number of popular visual neural network models optimised for the AI Camera in our model zoo, along with visualisation example scripts using Picamera2.
Which product should I buy?
Should you buy a Raspberry Pi AI Kit, or a Raspberry Pi AI Camera? The AI Kit has higher theoretical performance than the AI Camera, and can support a broader range of models, but is only compatible with Raspberry Pi 5. The AI Camera is more compact, has a lower total cost if you don’t already own a camera, and is compatible with all models of Raspberry Pi.
Ultimately, both products provide great acceleration performance for common models, and both have been optimised to work smoothly with our camera software stack.
Getting started and going further
Check out our Getting Started Guide. There you’ll find instructions on installing the AI Camera hardware, setting up the software environment, and running the examples and neural networks in our model zoo.
Sony’s AITRIOS Developer site has more technical resources on the IMX500 sensor, in particular the IMX500 Converter and IMX500 Package documentation, which will be useful for users who want to run custom-trained networks on the AI Camera.
We’ve been inspired by the incredible AI projects you’ve built over the years with Raspberry Pi, and your hard work and inventiveness encourages us to invest in the tools that will help you go further. The arrival of first the AI Kit, and now the AI Camera, opens up a whole new world of opportunities for high-resolution, high-frame rate, high-quality visual AI: we don’t know what you’re going to build with them, but we’re sure it will be awesome.
Expanding the open-source Meta Llama collection of models, the Llama 3.2 collection includes vision language models (VLMs), small language models (SLMs), and an...
Expanding the open-source Meta Llama collection of models, the Llama 3.2 collection includes vision language models (VLMs), small language models (SLMs), and an updated Llama Guard model with support for vision. When paired with the NVIDIA accelerated computing platform, Llama 3.2 offers developers, researchers, and enterprises valuable new capabilities and optimizations to realize their…
Vision-language models (VLMs) combine the powerful language understanding of foundational LLMs with the vision capabilities of vision transformers (ViTs) by...
Vision-language models (VLMs) combine the powerful language understanding of foundational LLMs with the vision capabilities of vision transformers (ViTs) by projecting text and images into the same embedding space. They can take unstructured multimodal data, reason over it, and return the output in a structured format. Building on a broad base of pretraining, they can be easily adapted for…
This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.
Using new tools on the command line can be frustrating. Even if we are confident that we’ve found the right tool, we might not know how to use it.
Telling an agent to RT(F)M
A typical workflow might look something like the following.
Install tool.
Read the documentation.
Run the command.
Repeat.
Can we improve this flow using LLMs?
Install tool
Docker provides us with isolated environments to run tools. Instead of requiring that commands be installed, we have created minimal Docker images for each tool so that using the tool does not impact the host system. Leave no trace, so to speak.
Read the documentation
Man pages are one of the ways that authors of tools ship content about how to use that tool. This content also comes with standard retrieval mechanisms (the man tool). A tool might also support a command-line option like --help. Let’s start with the idealistic notion that we should be able to retrieve usage information from the tool itself.
In this experiment, we’ve created two entry points for each tool. The first entry point is the obvious one. It is a set of arguments passed directly to a command-line program. The OpenAI-compatible description that we generate for this entry point is shown below. We are using the same interface for every tool.
{"name": "run_my_tool",
"description": "Run the my_tool command.",
"parameters":
{"type": "object",
"properties":
{"args":
{"type": "string",
"description": "The arguments to pass to my_tool"}}},
"container": {"image": "namespace/my_tool:latest"}}
The second entrypoint gives the agent the ability to read the man page and, hopefully, improve its ability to run the first entrypoint. The second entrypoint is simpler, because it only does one thing (asks a tool how to use it).
{"name": "my_tool_manual",
"description": "Read the man page for my_tool",
"container": {"image": "namespace/my_tool:latest", "command": ["man"]}}
Run the command
Let’s start with a simple example. We want to use a tool called qrencode to generate a QR code for a link. We have used our image generation pipeline to package this tool into a minimal image for qrencode. We will now pass this prompt to a few different LLMs; we are using LLMs that have been trained for tool calling (e.g., GPT 4, Llama 3.1, and Mistral). Here’s the prompt that we are testing:
Generate a QR code for the content https://github.com/docker/labs-ai-tools-for-devs/blob/main/prompts/qrencode/README.md. Save the generated image to qrcode.png.
If the command fails, read the man page and try again.
Note the optimism in this prompt. Because it’s hard to predict what different LLMs have already seen in their training sets, and many command-line tools use common names for arguments, it’s interesting to see what LLM will infer before adding the man page to the context.
The output of the prompt is shown below. Grab your phone and check it out.
Figure 1: Content QR code generated by AI assistant.
Repeat
When an LLM generates a description of how to run something, it will usually format that output in such a way that it will be easy for a user to cut and paste the response into a terminal:
qrencode -o qrcode.png 'my content'
However, if the LLM is generating tool calls, we’ll see output that is instead formatted to be easier to run:
We respond to this by spinning up a Docker container.
Running the tool as part of the conversation loop is useful even when the command fails. In Unix, there are standard ways to communicate failures. For example, we have exit codes, and stderr streams. This is how tools create feedback loops and correct our behavior while we’re iterating at the terminal. This same mechanism can be used in a conversation loop involving an LLM.
To illustrate, here is another simple example. We’ll try running a tool that writes ASCII art with the following prompt.
Write ascii art with the message "Docker" using the font "helvetica". Read the man page for figlet if you detect any errors and then try again.
In our test, this did fail. However, it also described the apparent issue on the stderr output stream.
Error: error: could not load font helvetica
By including this message in the conversation loop, the assistant can suggest different courses of action. Different LLMs produced different results here. For example, Llama 3.1 gives instructions for how to install the missing font. On the other hand, GPT 4 re-ran the command, but only after having made the “executive” decision to try a different font.
I'm sorry, it seems the font Helvetica is not available. Let's try it with the default font.
Here is the output of the figlet command with the default font (term):
____ _
| \ _ | | _____ _
| | | |/ \ / _| |/ / \ '_|
| |_| | (_) | (__| < __/ |
|____/ \___/ \___|_|\_\___|_|
We are very early in understanding how to take advantage of this apparent capacity to try different approaches. But this is another reason why quarantining these tools in Docker containers is useful. It limits their blast radius while we encourage experimentation.
Results
We started by creating a pipeline to produce minimal Docker images for each tool. The set of tools was selected based on whether they have outputs useful for developer-facing workflows. We continue to add new tools as we think of new use cases. The initial set is listed below.
There was a set of initial problems with context extraction.
Missing manual pages
Only about 60% of the tools we selected have man pages. However, even in those cases, there are usually other ways to get help content. The following steps show the final procedure we used:
Try to run the man page.
Try to run the tool with the argument --help.
Try to run the tool with the argument -h.
Try to run the tool with --broken args and then read stderr.
Using this procedure, every tool in the list above eventually produced documentation.
Long manual pages
Limited context lengths impacted some of the longer manual pages, so it was still necessary to employ standard RAG techniques to summarize verbose man pages. Our tactic was to focus on descriptions of command-line arguments and sections that had sample usage. These had the largest impact on the quality of the agent’s output. The structure of Unix man pages helped with the chunking, because we were able to rely on standard sections to chunk the content.
Subcommands
For a small set of tools, it was necessary to traverse a tree of help menus. However, these were all relatively popular tools, and the LLMs we deployed already knew about this command structure. It’s easy to check this out for yourself. Ask an LLM, for example: “What are the subcommands of Git?” or “What are the subcommands of Docker?” Maybe only popular tools get big enough that they start to be broken up into subcommands.
Summary
We should consider the active role that agents can play when determining how to use a tool. The Unix model has given us standards such as man pages, stderr streams, and exit codes, and we can take advantage of these conventions when asking an assistant to learn a tool. Beyond distribution, Docker also provides us with process isolation, which is useful when creating environments for safe exploration.
Whether or not an AI can successfully generate tool calls may also become a metric for whether or not a tool has been well documented.
Login
