If you’ve ever been tangled in the complexities of setting up a WordPress environment, you’re not alone. WordPress powers more than 40% of all websites, making it the world’s most popular content management system (CMS). Its versatility is unmatched, but traditional local development setups like MAMP, WAMP, or XAMPP can lead to inconsistencies and the infamous “it works on my machine” problem.

As projects scale and teams grow, the need for a consistent, scalable, and efficient development environment becomes critical. That’s where Docker comes into play, revolutionizing how we develop and deploy WordPress sites. To make things even smoother, we’ll integrate Traefik, a modern reverse proxy that automatically obtains TLS certificates, ensuring that your site runs securely over HTTPS. Traefik is available as a Docker Official Image from Docker Hub.

In this comprehensive guide, I’ll show how to Dockerize your WordPress site using real-world examples. We’ll dive into creating Dockerfiles, containerizing existing WordPress instances — including migrating your data — and setting up Traefik for automatic TLS certificates. Whether you’re starting fresh or migrating an existing site, this tutorial has you covered.

Let’s dive in!

Why should you containerize your WordPress site?

Containerizing your WordPress site offers a multitude of benefits that can significantly enhance your development workflow and overall site performance.

Increased page load speed

Docker containers are lightweight and efficient. By packaging your application and its dependencies into containers, you reduce overhead and optimize resource usage. This can lead to faster page load times, improving user experience and SEO rankings.

Efficient collaboration and version control

With Docker, your entire environment is defined as code. This ensures that every team member works with the same setup, eliminating environment-related discrepancies. Version control systems like Git can track changes to your Dockerfiles and to wordpress-traefik-letsencrypt-compose.yml, making collaboration seamless.

Easy scalability

Scaling your WordPress site to handle increased traffic becomes straightforward with Docker and Traefik. You can spin up multiple Docker containers of your application, and Traefik will manage load balancing and routing, all while automatically handling TLS certificates.

Simplified environment setup

Setting up your development environment becomes as simple as running a few Docker commands. No more manual installations or configurations — everything your application needs is defined in your Docker configuration files.

Simplified updates and maintenance

Updating WordPress or its dependencies is a breeze. Update your Docker images, rebuild your containers, and you’re good to go. Traefik ensures that your routes and certificates are managed dynamically, reducing maintenance overhead.

Getting started with WordPress, Docker, and Traefik

Before we begin, let’s briefly discuss what Docker and Traefik are and how they’ll revolutionize your WordPress development workflow.

• Docker is a cloud-native development platform that simplifies the entire software development lifecycle by enabling developers to build, share, test, and run applications in containers. It streamlines the developer experience while providing built-in security, collaboration tools, and scalable solutions to improve productivity across teams.
• Traefik is a modern reverse proxy and load balancer designed for microservices. It integrates seamlessly with Docker and can automatically obtain and renew TLS certificates from Let’s Encrypt.

How long will this take?

Setting up this environment might take around 45-60 minutes, especially if you’re integrating Traefik for automatic TLS certificates and migrating an existing WordPress site.

Documentation links

• Docker documentation
• Traefik documentation
• WordPress Docker Bitnami image
• Docker Compose documentation

Tools you’ll need

• Docker Desktop: If you don’t already have the latest version installed, download and install Docker Desktop.
• A domain name: Required for Traefik to obtain TLS certificates from Let’s Encrypt.
• Access to DNS settings: To point your domain to your server’s IP address.
• Code editor: Your preferred code editor for editing configuration files.
• Command-line interface (CLI): Access to a terminal or command prompt.
• Existing WordPress data: If you’re containerizing an existing site, ensure you have backups of your WordPress files and MySQL database.

What’s the WordPress Docker Bitnami image?

To simplify the process, we’ll use the Bitnami WordPress image from Docker Hub, which comes pre-packaged with a secure, optimized environment for WordPress. This reduces configuration time and ensures your setup is up to date with the latest security patches.

Using the Bitnami WordPress image streamlines your setup process by:

• Simplifying configuration: Bitnami images come with sensible defaults and configurations that work out of the box, reducing the time spent on setup.
• Enhancing security: The images are regularly updated to include the latest security patches, minimizing vulnerabilities.
• Ensuring consistency: With a standardized environment, you avoid the “it works on my machine” problem and ensure consistency across development, staging, and production.
• Including additional tools: Bitnami often includes helpful tools and scripts for backups, restores, and other maintenance tasks.

By choosing the Bitnami WordPress image, you can leverage a tested and optimized environment, reducing the risk of configuration errors and allowing you to focus more on developing your website.

Key features of Bitnami WordPress Docker image:

• Optimized for production: Configured with performance and security in mind.
• Regular updates: Maintained to include the latest WordPress version and dependencies.
• Ease of use: Designed to be easy to deploy and integrate with other services, such as databases and reverse proxies.
• Comprehensive documentation: Offers guides and support to help you get started quickly.

Why we use Bitnami in the examples:

In our Docker Compose configurations, we specified:

WORDPRESS_IMAGE_TAG=bitnami/wordpress:6.3.1

This indicates that we’re using the Bitnami WordPress image, version 6.3.1. The Bitnami image aligns well with our goals for a secure, efficient, and easy-to-manage WordPress environment, especially when integrating with Traefik for automatic TLS certificates.

By leveraging the Bitnami WordPress Docker image, you’re choosing a robust and reliable foundation for your WordPress projects. This approach allows you to focus on building great websites without worrying about the underlying infrastructure.

How to Dockerize an existing WordPress site with Traefik

Let’s walk through dockerizing your WordPress site using practical examples, including your .env and wordpress-traefik-letsencrypt-compose.yml configurations. We’ll also cover how to incorporate your existing data into the Docker containers.

Step 1: Preparing your environment variables

First, create a .env file in the same directory as your wordpress-traefik-letsencrypt-compose.yml file. This file will store all your environment variables.

Example .env file:

# Traefik Variables
TRAEFIK_IMAGE_TAG=traefik:2.9
TRAEFIK_LOG_LEVEL=WARN
TRAEFIK_ACME_EMAIL=your-email@example.com
TRAEFIK_HOSTNAME=traefik.yourdomain.com
# Basic Authentication for Traefik Dashboard
# Username: traefikadmin
# Passwords must be encoded using BCrypt https://hostingcanada.org/htpasswd-generator/
TRAEFIK_BASIC_AUTH=traefikadmin:$$2y$$10$$EXAMPLEENCRYPTEDPASSWORD

# WordPress Variables
WORDPRESS_MARIADB_IMAGE_TAG=mariadb:11.4
WORDPRESS_IMAGE_TAG=bitnami/wordpress:6.6.2
WORDPRESS_DB_NAME=wordpressdb
WORDPRESS_DB_USER=wordpressdbuser
WORDPRESS_DB_PASSWORD=your-db-password
WORDPRESS_DB_ADMIN_PASSWORD=your-db-admin-password
WORDPRESS_TABLE_PREFIX=wpapp_
WORDPRESS_BLOG_NAME=Your Blog Name
WORDPRESS_ADMIN_NAME=AdminFirstName
WORDPRESS_ADMIN_LASTNAME=AdminLastName
WORDPRESS_ADMIN_USERNAME=admin
WORDPRESS_ADMIN_PASSWORD=your-admin-password
WORDPRESS_ADMIN_EMAIL=admin@yourdomain.com
WORDPRESS_HOSTNAME=wordpress.yourdomain.com
WORDPRESS_SMTP_ADDRESS=smtp.your-email-provider.com
WORDPRESS_SMTP_PORT=587
WORDPRESS_SMTP_USER_NAME=your-smtp-username
WORDPRESS_SMTP_PASSWORD=your-smtp-password

Notes:

• Replace placeholder values (e.g., your-email@example.com, your-db-password) with your actual credentials.
• Do not commit this file to version control if it contains sensitive information.
• Use a password encryption tool to generate the encrypted password for TRAEFIK_BASIC_AUTH. For example, you can use the htpasswd generator.

Step 2: Creating the Docker Compose file

Create a wordpress-traefik-letsencrypt-compose.yml file that defines your services, networks, and volumes. This YAML file is crucial for configuring your WordPress installation through Docker.

Example wordpress-traefik-letsencrypt-compose.yml.

networks:
  wordpress-network:
    external: true
  traefik-network:
    external: true

volumes:
  mariadb-data:
  wordpress-data:
  traefik-certificates:

services:
  mariadb:
    image: ${WORDPRESS_MARIADB_IMAGE_TAG}
    volumes:
      - mariadb-data:/var/lib/mysql
    environment:
      MARIADB_DATABASE: ${WORDPRESS_DB_NAME}
      MARIADB_USER: ${WORDPRESS_DB_USER}
      MARIADB_PASSWORD: ${WORDPRESS_DB_PASSWORD}
      MARIADB_ROOT_PASSWORD: ${WORDPRESS_DB_ADMIN_PASSWORD}
    networks:
      - wordpress-network
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 60s
    restart: unless-stopped

  wordpress:
    image: ${WORDPRESS_IMAGE_TAG}
    volumes:
      - wordpress-data:/bitnami/wordpress
    environment:
      WORDPRESS_DATABASE_HOST: mariadb
      WORDPRESS_DATABASE_PORT_NUMBER: 3306
      WORDPRESS_DATABASE_NAME: ${WORDPRESS_DB_NAME}
      WORDPRESS_DATABASE_USER: ${WORDPRESS_DB_USER}
      WORDPRESS_DATABASE_PASSWORD: ${WORDPRESS_DB_PASSWORD}
      WORDPRESS_TABLE_PREFIX: ${WORDPRESS_TABLE_PREFIX}
      WORDPRESS_BLOG_NAME: ${WORDPRESS_BLOG_NAME}
      WORDPRESS_FIRST_NAME: ${WORDPRESS_ADMIN_NAME}
      WORDPRESS_LAST_NAME: ${WORDPRESS_ADMIN_LASTNAME}
      WORDPRESS_USERNAME: ${WORDPRESS_ADMIN_USERNAME}
      WORDPRESS_PASSWORD: ${WORDPRESS_ADMIN_PASSWORD}
      WORDPRESS_EMAIL: ${WORDPRESS_ADMIN_EMAIL}
      WORDPRESS_SMTP_HOST: ${WORDPRESS_SMTP_ADDRESS}
      WORDPRESS_SMTP_PORT: ${WORDPRESS_SMTP_PORT}
      WORDPRESS_SMTP_USER: ${WORDPRESS_SMTP_USER_NAME}
      WORDPRESS_SMTP_PASSWORD: ${WORDPRESS_SMTP_PASSWORD}
    networks:
      - wordpress-network
      - traefik-network
    healthcheck:
      test: timeout 10s bash -c ':> /dev/tcp/127.0.0.1/8080' || exit 1
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 90s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.wordpress.rule=Host(`${WORDPRESS_HOSTNAME}`)"
      - "traefik.http.routers.wordpress.service=wordpress"
      - "traefik.http.routers.wordpress.entrypoints=websecure"
      - "traefik.http.services.wordpress.loadbalancer.server.port=8080"
      - "traefik.http.routers.wordpress.tls=true"
      - "traefik.http.routers.wordpress.tls.certresolver=letsencrypt"
      - "traefik.http.services.wordpress.loadbalancer.passhostheader=true"
      - "traefik.http.routers.wordpress.middlewares=compresstraefik"
      - "traefik.http.middlewares.compresstraefik.compress=true"
      - "traefik.docker.network=traefik-network"
    restart: unless-stopped
    depends_on:
      mariadb:
        condition: service_healthy
      traefik:
        condition: service_healthy

  traefik:
    image: ${TRAEFIK_IMAGE_TAG}
    command:
      - "--log.level=${TRAEFIK_LOG_LEVEL}"
      - "--accesslog=true"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--ping=true"
      - "--ping.entrypoint=ping"
      - "--entryPoints.ping.address=:8082"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--providers.docker=true"
      - "--providers.docker.endpoint=unix:///var/run/docker.sock"
      - "--providers.docker.exposedByDefault=false"
      - "--certificatesresolvers.letsencrypt.acme.tlschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.email=${TRAEFIK_ACME_EMAIL}"
      - "--certificatesresolvers.letsencrypt.acme.storage=/etc/traefik/acme/acme.json"
      - "--metrics.prometheus=true"
      - "--metrics.prometheus.buckets=0.1,0.3,1.2,5.0"
      - "--global.checkNewVersion=true"
      - "--global.sendAnonymousUsage=false"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - traefik-certificates:/etc/traefik/acme
    networks:
      - traefik-network
    ports:
      - "80:80"
      - "443:443"
    healthcheck:
      test: ["CMD", "wget", "http://localhost:8082/ping","--spider"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 5s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_HOSTNAME}`)"
      - "traefik.http.routers.dashboard.service=api@internal"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.services.dashboard.loadbalancer.server.port=8080"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=letsencrypt"
      - "traefik.http.services.dashboard.loadbalancer.passhostheader=true"
      - "traefik.http.routers.dashboard.middlewares=authtraefik"
      - "traefik.http.middlewares.authtraefik.basicauth.users=${TRAEFIK_BASIC_AUTH}"
      - "traefik.http.routers.http-catchall.rule=HostRegexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    restart: unless-stopped

Notes:

• Networks: We’re using external networks (wordpress-network and traefik-network). We’ll create these networks before deploying.
• Volumes: Volumes are defined for data persistence.
• Services: We’ve defined mariadb, wordpress, and traefik services with the necessary configurations.
• Health checks: Ensure that services are healthy before dependent services start.
• Labels: Configure Traefik routing, HTTPS settings, and enable the dashboard with basic authentication.

Step 3: Creating external networks

Before deploying your Docker Compose configuration, you need to create the external networks specified in your wordpress-traefik-letsencrypt-compose.yml.

Run the following commands to create the networks:

docker network create traefik-network
docker network create wordpress-network

Step 4: Deploying your WordPress site

Deploy your WordPress site using Docker Compose with the following command (Figure 1):

docker compose -f wordpress-traefik-letsencrypt-compose.yml -p website up -d

Explanation:

• -f wordpress-traefik-letsencrypt-compose.yml: Specifies the Docker Compose file to use.
• -p website: Sets the project name to website.
• up -d: Builds, (re)creates, and starts containers in detached mode.

Step 5: Verifying the deployment

Check that all services are running (Figure 2):

docker ps

You should see the mariadb, wordpress, and traefik services up and running.

Step 6: Accessing your WordPress site and Traefik dashboard

WordPress site: Navigate to https://wordpress.yourdomain.com in your browser. Type in the username and password you set earlier in the .env file and click the Log In button. You should see your WordPress site running over HTTPS, with a valid TLS certificate automatically obtained by Traefik (Figure 3).

Important: To get cryptographic certificates, you need to set up A-type records in your external DNS zone that point to your server’s IP address where Traefik is installed. If you’ve just set up these records, wait a bit before starting the service installation because it can take anywhere from a few minutes to 48 hours — sometimes even longer — for these changes to fully spread across DNS servers.

• Traefik dashboard: Access the Traefik dashboard at https://traefik.yourdomain.com. You’ll be prompted for authentication. Use the username and password specified in your .env file (Figure 4).

Step 7: Incorporating your existing WordPress data

If you’re migrating an existing WordPress site, you’ll need to incorporate your existing files and database into the Docker containers.

Step 7.1: Restoring WordPress files

Copy your existing WordPress files into the wordpress-data volume.

Option 1: Using Docker volume mapping

Modify your wordpress-traefik-letsencrypt-compose.yml to map your local WordPress files directly:

volumes:
  - ./your-wordpress-files:/bitnami/wordpress

Option 2: Copying files into the running container

Assuming your WordPress backup is in ./wordpress-backup, run:

docker cp ./wordpress-backup/. wordpress_wordpress_1:/bitnami/wordpress/

Step 7.2: Importing your database

Export your existing WordPress database using mysqldump or phpMyAdmin.

Example:

mysqldump -u your_db_user -p your_db_name > wordpress_db_backup.sql

Copy the database backup into the MariaDB container:

docker cp wordpress_db_backup.sql wordpress_mariadb_1:/wordpress_db_backup.sql

Access the MariaDB container:

docker exec -it wordpress_mariadb_1 bash

Import the database:

mysql -u root -p${WORDPRESS_DB_ADMIN_PASSWORD} ${WORDPRESS_DB_NAME} < wordpress_db_backup.sql

Step 7.3: Update wp-config.php (if necessary)

Because we’re using environment variables, WordPress should automatically connect to the database. However, if you have custom configurations, ensure they match the settings in your .env file.

Note: The Bitnami WordPress image manages wp-config.php automatically based on environment variables. If you need to customize it further, you can create a custom Dockerfile.

Step 8: Creating a custom Dockerfile (optional)

If you need to customize the WordPress image further, such as installing additional PHP extensions or modifying configuration files, create a Dockerfile in your project directory.

Example Dockerfile:

# Use the Bitnami WordPress image as the base
FROM bitnami/wordpress:6.3.1

# Install additional PHP extensions if needed
# RUN install_packages php7.4-zip php7.4-mbstring

# Copy custom wp-content (if not using volume mapping)
# COPY ./wp-content /bitnami/wordpress/wp-content

# Set working directory
WORKDIR /bitnami/wordpress

# Expose port 8080
EXPOSE 8080

Build the custom image:

Modify your wordpress-traefik-letsencrypt-compose.yml to build from the Dockerfile:

wordpress:
  build: .
  # Rest of the configuration

Then, rebuild your containers:

docker compose -p wordpress up -d --build

Step 9: Customizing WordPress within Docker

Adding themes and plugins

Because we’ve mapped the wordpress-data volume, any changes you make within the WordPress container (like installing plugins or themes) will persist across container restarts.

• Via WordPress admin dashboard: Install themes and plugins as you normally would through the WordPress admin interface (Figure 5).

• Manually: Access the container and place your themes or plugins directly.

Example:

docker exec -it wordpress_wordpress_1 bash
cd /bitnami/wordpress/wp-content/themes
# Add your theme files here

Managing and scaling WordPress with Docker and Traefik

Scaling your WordPress service

To handle increased traffic, you might want to scale your WordPress instances.

docker compose -p wordpress up -d --scale wordpress=3

Traefik will automatically detect the new instances and load balance traffic between them.

Note: Ensure that your WordPress setup supports scaling. You might need to externalize session storage or use a shared filesystem for media uploads.

Updating services

To update your services to the latest images:

Pull the latest images:

docker compose -p wordpress pull

Recreate containers:

docker compose -p wordpress up -d

Monitoring and logs

Docker logs:
View logs for a specific service:

docker compose -p wordpress logs -f wordpress

Traefik dashboard:
Use the Traefik dashboard to monitor routing, services, and health checks.

Optimizing your WordPress Docker setup

Implementing caching with Redis

To improve performance, you can add Redis for object caching.

Update wordpress-traefik-letsencrypt-compose.yml:

services:
  redis:
    image: redis:alpine
    networks:
      - wordpress-network
    restart: unless-stopped

Configure WordPress to use Redis:

• Install a Redis caching plugin like Redis Object Cache.
• Configure it to connect to the redis service.

Security best practices

• Secure environment variables:
  • Use Docker secrets or environment variables to manage sensitive information securely.
  • Avoid committing sensitive data to version control.
• Restrict access to Docker socket:
  • The Docker socket is mounted read-only (:ro) to minimize security risks.
• Keep images updated:
  • Regularly update your Docker images to include security patches and improvements.

Advanced Traefik configurations

• Middleware: Implement middleware for rate limiting, IP whitelisting, and other request transformations.
• Monitoring: Integrate with monitoring tools like Prometheus and Grafana for advanced insights.
• Wildcard certificates: Configure Traefik to use wildcard certificates if you have multiple subdomains.

Wrapping up

Dockerizing your WordPress site with Traefik simplifies your development and deployment processes, offering consistency, scalability, and efficiency. By leveraging practical examples and incorporating your existing data, we’ve created a tailored guide to help you set up a robust WordPress environment.

Whether you’re managing an existing site or starting a new project, this setup empowers you to focus on what you do best — developing great websites — while Docker and Traefik handle the heavy lifting.

So go ahead, give it a shot! Embracing these tools is a step toward modernizing your workflow and staying ahead in the ever-evolving tech landscape.

Learn more

To further enhance your skills and optimize your setup, check out these resources:

• Subscribe to the Docker Newsletter
• Get the latest release of Docker Desktop
• Docker documentation
• Compose file reference
• WordPress Docker Bitnami image
• Traefik documentation
• Understand Docker networking
• Manage sensitive data with Docker secrets
• WordPress plugins for Docker
• Redis object cache plugin
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

Key features of the Docker Desktop 4.35 release include: 

• Secure development environments with organization access tokens (Beta) 
• Docker Home enhances your Docker experience (Beta) 
• Accelerate development with a terminal in Docker Desktop (GA)  
• Volumes Export (GA)
• Improved performance on macOS
• Docker Desktop for Red Hat Enterprise Linux

Organization access tokens (Beta) 

Before the beta release of organization access tokens, managing developer access to Docker resources was challenging, as it relied heavily on individual user accounts, leading to security risks and administrative inefficiencies. 

Organization access tokens let you manage access at the organizational level, providing enhanced security. This feature allows teams to operate more securely and efficiently with centralized user management, reduced administrative overhead, and the flexibility to scale access as the organization grows. For businesses, this feature offers significant value by improving governance, enhancing security, and supporting scalable infrastructure from an administrative perspective. 

Organizational access tokens empower organizations to maintain tighter control over their resources and security, making Docker Desktop even more valuable for enterprise users. This is one piece of the continuous updates we’re releasing to support administrators across large enterprise companies, ensuring they have the tools needed to manage complex environments with efficiency and confidence.

Docker Home (Beta) 

Sign in to your Docker account to see the release of the new Docker Home page (Figure 1). The new Docker Home marks a milestone in Docker’s journey as a multi-product company, reinforcing Docker’s commitment to providing an expanding suite of solutions that help developers and businesses containerize applications with ease.

• Unified experience: The home page provides a central hub for users to access Docker products, manage subscriptions, adjust settings, and find resources — all in one place. This approach simplifies navigation for developers and admins.
• Admin access: Administrators can manage organizations, users, and onboarding processes through the new portal, with access to dashboards for monitoring Docker usage.
• Future enhancements: Future updates will add personalized features for different roles, and business subscribers will gain access to tools like the Docker Support portal and organization-wide notifications.

Terminal experience in Docker Desktop

Our terminal feature in Docker Desktop is now generally available. While managing containerized applications, developers have often faced friction and inefficiencies when switching between the Docker Desktop CLI and GUI. This constant context switching disrupted workflows and reduced productivity. 

The terminal enhancement integrates a terminal directly within the Docker Desktop GUI, enabling seamless transitions between CLI and GUI interactions within a single window. By incorporating a terminal shell into the Docker Desktop interface (Figure 2), we significantly reduce the friction associated with context switching for developers.

This functionality is designed to streamline workflows, accelerate delivery times, and enhance overall developer productivity.

Volumes Export is GA 

With the 4.35 release, we’ve elevated volume backup capabilities in Docker Desktop, introducing an upgraded feature set (Figure 3). This enhancement directly integrates the previous Volumes Backup & Share extension directly into Docker Desktop, streamlining your backup processes.

Although this release marks a significant step forward, it’s just the beginning. We’re committed to expanding these capabilities, adding even more value in future updates. Check out the beta of Scheduled Backups as well as External Cloud Storage backups, which are also available. 

Significantly improved performance experience on macOS (Beta)

Docker Desktop 4.35 also includes a beta release of Docker VMM, a container-optimized hypervisor for Apple Silicon Macs. Local developer workflows rely heavily on the performance of the hypervisor layer for everything from handling individual timer interrupts to accessing files and downloading images from the network. 

Docker VMM allows us to optimize the Linux kernel and hypervisor layer together, massively improving the speed of many common developer tasks. For example, iterating over a large shared file system with find is now 2x faster than on Docker Desktop 4.34 with a cold cache and up to 25x faster — faster than running natively on the Mac — when the cache is warm. This is only the beginning. Thanks to Docker VMM, we have many exciting new performance improvements in the pipeline.

Enable Docker VMM via Settings > General > Virtual Machine options and try it for your developer workflows today (Figure 4).

Docker Desktop for Red Hat Enterprise Linux 

Today we are excited to announce the general availability of Docker Desktop for Red Hat Enterprise Linux (RHEL). This feature marks a great milestone for both Docker and our growing community of developers.

By making Docker Desktop available on RHEL, we’re not only extending our reach — we’re meeting developers where they are. RHEL users can now access a seamless containerized development experience directly on the same OS that might power their production environments.

Docker Desktop for RHEL (Figure 5) offers the same intuitive interface, integrated tooling, and performance optimizations that you’ve come to expect on the other supported Linux distributions.

How to install Docker Desktop on Red Hat Enterprise Linux

Download links and information can be found in our release notes. 

Looking for support?

Did you know that you can get Premium Customer Support for Docker Desktop with a Pro or Team subscription? With this GA release, we’re now ready to officially help support you if you’re thinking about using Docker Desktop. Check out our pricing page to learn more about what’s included in a Pro or Team subscription, and if it’s right for you.

Explore the latest updates

With this latest wave of updates, from the security enhancements of organization access tokens to the performance boost of Docker VMM for Apple Silicon Macs, we’re pushing Docker Desktop forward to meet the evolving needs of developers and organizations alike. Each new feature is designed to make development smoother, faster, and more secure — whether you’re managing large teams or optimizing your individual workflow. 

We’re continuing to make improvements, with more tools and features on the way to help you build, manage, and scale your projects efficiently. Explore the latest updates and see how they can enhance your development experience

Learn more

• Subscribe to the Docker Newsletter.
• Authenticate and update to receive your subscription level’s newest Docker Desktop features.
• New to Docker? Create an account.
• Learn more about host networking support.
• Have questions? The Docker community is here to help.

Docker Desktop is more than just a local application for containerized development — it’s your gateway to an integrated suite of cloud-native tools that streamline the entire development workflow. While Docker Desktop can be used without signing in, doing so unlocks the full potential of Docker’s powerful, interconnected ecosystem. By signing in, you gain access to advanced features and services across Docker Hub, Build Cloud, Scout, and Testcontainers Cloud, enabling deeper collaboration, enhanced security insights, and scalable cloud resources. 

This blog post explores the full range of capabilities unlocked by signing in to Docker Desktop, connecting you to Docker’s integrated suite of cloud-native development tools. From enhanced security insights with Docker Scout to scalable build and testing resources through Docker Build Cloud and Testcontainers Cloud, signing in allows developers and administrators to fully leverage Docker’s unified platform.

Note that the following sections refer to specific Docker subscription plans. With Docker’s newly streamlined subscription plans — Docker Personal, Docker Pro, Docker Team, and Docker Business — developers and organizations can access a scalable suite of tools, from individual productivity boosters to enterprise-grade governance and security. Visit the Docker pricing page to learn more about how these plans support different team sizes and workflows. 

Benefits for developers when logged in

Docker Personal

• Access to private repositories: Unlock secure collaboration through private repositories on Docker Hub, ensuring that your sensitive code and dependencies are managed securely across teams and projects.
• Increased pull rate: Boost your productivity with an increased pull rate from Docker Hub (40 pulls/hour per user), ensuring smoother, uninterrupted development workflows without waiting on rate limits. The rate limit without authentication is 10 pulls/hour per IP.
• Docker Scout CLI: Leverage Docker Scout to proactively secure your software supply chain with continuous security insights from code to production. By signing in, you gain access to powerful CLI commands that help prevent vulnerabilities before they reach production. 
• Build Cloud and Testcontainers Cloud: Experience the full power of Docker Build Cloud and Testcontainers Cloud with free trials (7-day for Build Cloud, 30-day for Testcontainers Cloud). These trials give you access to scalable cloud infrastructure that speeds up image builds and enables more reliable integration testing.

Docker Pro/Team/Business 

For users with a paid Docker subscription, additional features are unlocked.

• Unlimited pull rate: No Hub rate limit will be enforced for users with a paid subscription plan. 
• Docker Scout base image recommendations: Docker Scout offers continuous recommendations for base image updates, empowering developers to secure their applications at the foundational level and fix vulnerabilities early in the development lifecycle.

• Docker Debug: The docker debug CLI command can help you debug containers, while the images contain the minimum required to run your application.

Docker Debug functionalities have also been integrated into the container view of the Docker Desktop UI.

• Synchronized file shares: Host to Docker Desktop VM file sharing via bind mounts can be quite slow for large codebases. Speed up your development cycle with synchronized file shares, allowing you to sync large codebases into containers quickly and efficiently without performance bottlenecks—helping developers iterate faster on critical projects.

• Additional free minutes for Docker Build Cloud: Docker Build Cloud helps developer teams speed up image builds by offloading the build process to the cloud. The following benefits are available for users depending on the subscription plan. 
  • Docker Pro: 200 mins/month per org
  • Docker Team: 500 mins/month per org
  • Docker Business: 1500 mins/month per org

• Additional free minutes for Testcontainers Cloud: Testcontainers Cloud simplifies the process for developers to run reliable integration tests using real dependencies defined in code, whether on their laptops or within their team’s CI pipeline. Depending on the subscription plan, the following benefits are available for users:
  • Docker Pro: 100 mins/month per org
  • Docker Team: 500 mins/month per org
  • Docker Business: 1,500 mins/month per org

Benefits for administrators when your users are logged in

Docker Business

Security and governance

The Docker Business plan offers enterprise-grade security and governance controls, which are only applicable if users are signed in. As of Docker Desktop 4.35.0, these features include:

• Image Access Management and Registry Access Management: Enhance security governance by controlling exactly which images and registries your developers can access or push to, ensuring compliance with organizational security policies and minimizing risk exposure.
• Settings management: Enforce organization-wide security policies by locking default settings across Docker Desktop instances, ensuring consistent configuration and compliance throughout your team’s development environment.
• Enhanced Container Isolation: Protect developer environments from malicious containers with rootless containers and much more.
• Air-gapped containers: Restrict network activity originating from containers.
• Private extensions marketplace: Limit Docker Desktop extensions that developers can install.
• Kerberos and NTLM authentication for proxies: Centralize Docker Desktop authentication to network proxies without prompts.
• SOCKS5 proxy support: Enable the usage of SOCKS5 proxies for Docker Desktop.

License management

Tracking usage for licensing purposes can be challenging for administrators due to Docker Desktop not requiring authentication by default. By ensuring all users are signed in, administrators can use Docker Hub’s organization members list to manage licenses effectively.

This can be coupled with Docker Business’s Single Sign-On and SCIM capabilities to ease this process further. 

Insights

Administrators and other stakeholders (such as engineering managers) must comprehensively understand Docker Desktop usage within their organization. With developers signed into Docker Desktop, admins gain actionable insights into usage, from feature adoption to image usage trends and login activity, helping administrators optimize team performance and security. A dashboard offering insights is now available to simplify monitoring. Contact your account rep to enable the dashboard.

Enforce sign-in for Docker Desktop

Docker Desktop includes a feature that allows administrators to require authentication at start-up. Admins can ensure that all developers sign in to access Docker Desktop, enabling full integration with Docker’s security and productivity features. Sign-in enforcement helps maintain continuous compliance with governance policies across the organization.

Developers can then click on the sign-in button, which takes them through the authentication flow. 

More information on how to enforce sign-in can be found in the documentation. 

Unlock the full potential of Docker’s integrated suite

Signing into Docker Desktop unlocks significant benefits for both developers and administrators, enabling teams to fully leverage Docker’s integrated, cloud-native suite. Whether improving productivity, securing the software supply chain, or enforcing governance policies, signing in maximizes the value of Docker’s unified platform — especially for organizations using Docker’s paid subscription plans.

Note that new features are introduced with each new release, so keep an eye on our blog and subscribe to the Docker Newsletter for the latest product and feature updates.

Up next

• Read Announcing Upgraded Docker Plans: Simpler, More Value, Better Development and Productivity.
• Learn more about the new Docker subscriptions.
• Get started with Docker Desktop.
• Subscribe to the Docker Newsletter.

This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.

How do you turn a PDF into a set of images? A modern suggestion could be to get Claude or GPT to write you a CLI command for something like ImageMagick. Let’s follow that train of thought.

If we enter a prompt like “How do I turn a PDF into a bunch of images using ImageMagick,” we will indeed get something that looks like it might be a solution.

Imagemagick includes a command-line tool called convert which you can use to convert PDF files into images. Here's how you can do it:

1. Open your terminal

2. Change directory to where your PDF is located:

```
cd /path/to/your/pdf

```
3. Use the following command to convert the PDF into images (in this case JPEG):

```
convert -density 300 -quality 100 your_file.pdf output_name.jpg

``` 

The agent will then explain how convert works and finish with a suggestion to “please make sure that you have installed ImageMagick before running these commands.”

Instead of worrying about whether ImageMagick is installed, however, let’s use Docker as our tool provider and prompt it to go ahead and run the command. If you’re following along in this series, you’ll have seen that we are using Markdown files to mix together tools and prompts.  Here’s the first prompt we tried:

---
tools:
  - name: imagemagick
---
# prompt user

Use Imagemagick to convert the family.pdf file into a bunch of jpg images.

After executing this prompt, the LLM generated a tool call, which we executed in the Docker runtime, and it successfully converted family.pdf into nine .jpg files (my family.pdf file had nine pages). 

Figure 1 shows the flow from our VSCode Extension.

We have given enough context to the LLM that it is able to plan a call to this ImageMagick binary. And, because this tool is available on Docker Hub, we don’t have to “make sure that ImageMagick is installed.” This would be the equivalent command if you were to use docker run directly:

# family.pdf must be located in your $PWD

docker run --rm -v $PWD:/project --workdir /project vonwig/imageMagick:latest convert -density 300 -quality 300 family.pdf family.jpg 

The tool ecosystem

How did this work? The process relied on two things:

• Tool distribution and discovery (pulling tools into Docker Hub for distribution to our Docker Desktop runtime).
• Automatic generation of Agent Tool interfaces.

When we first started this project, we expected that we’d begin with a small set of tools because the interface for each tool would take time to design. We thought we were going to need to bootstrap an ecosystem of tools that had been prepared to be used in these agent workflows. 

However, we learned that we can use a much more generic approach. Most tools already come with documentation, such as command-line help, examples, and man pages. Instead of treating each tool as something special, we are using an architecture where an agent responds to failures by reading documentation and trying again (Figure 2).

We see a process of experimenting with tools that is not unlike what we, as developers, do on the command line. Try a command line, read a doc, adjust the command line, and try again.

The value of this kind of looping has changed our expectations. Step one is simply pulling the tool into Docker Hub and seeing whether the agent can use it with nothing more than its out-of-the-box documentation. We are also pulling open source software (OSS)  tools directly from nixpkgs, which gives us access to tens of thousands of different tools to experiment with. 

Docker keeps our runtimes isolated from the host operating system, while the nixpkgs ecosystem and maintainers provide a rich source of OSS tools.

As expected, packaging agents still run into issues that force us to re-plan how tools are packaged. For example, the prompt we showed above might have generated the correct tool call on the first try, but the ImageMagick container failed on the first run with this terrible-looking error message:

function call failed call exited with non-zero code (1): Error: sh: 1: gs: not found  

Fortunately, feeding that error back into the LLM resulted in the suggestion that convert needs another tool, called Ghostscript, to run successfully. Our agent was not able to fix this automatically today. However, we adjusted the image build slightly and now the “latest” version of the vonwig/imagemagick:latest no longer has this issue. This is an example of something we only need to learn once.

The LLM figured out convert on its own. But its agency came from the addition of a tool.

Read the Docker Labs GenAI series to see more of what we’ve been working on.

Learn more

• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

When testing complex systems, the more edge cases you can identify, the better your software performs in the real world. But how do you efficiently generate hundreds or thousands of meaningful tests that reveal hidden bugs? Enter model-based testing (MBT), a technique that automates test case generation by modeling your software’s expected behavior.

In this demo, we’ll explore the model-based testing technique to perform regression testing on a simple REST API.

We’ll use the jqwik test engine on JUnit 5 to run property and model-based tests. Additionally, we’ll use Testcontainers to spin up Docker containers with different versions of our application.

Model-based testing

Model-based testing is a method for testing stateful software by comparing the tested component with a model that represents the expected behavior of the system. Instead of manually writing test cases, we’ll use a testing tool that:

• Takes a list of possible actions supported by the application
• Automatically generates test sequences from these actions, targeting potential edge cases
• Executes these tests on the software and the model, comparing the results

In our case, the actions are simply the endpoints exposed by the application’s API. For the demo’s code examples, we’ll use a basic service with a CRUD REST API that allows us to:

• Find an employee by their unique employee number
• Update an employee’s name
• Get a list of all the employees from a department
• Register a new employee

Once everything is configured and we finally run the test, we can expect to see a rapid sequence of hundreds of requests being sent to the two stateful services:

Docker Compose

Let’s assume we need to switch the database from Postgres to MySQL and want to ensure the service’s behavior remains consistent. To test this, we can run both versions of the application, send identical requests to each, and compare the responses.

We can set up the environment using a Docker Compose that will run two versions of the app:

• Model (mbt-demo:postgres): The current live version and our source of truth.
• Tested version (mbt-demo:mysql): The new feature branch under test.

services:
  ## MODEL
  app-model:
      image: mbt-demo:postgres
      # ...
      depends_on:
          - postgres
  postgres:
      image: postgres:16-alpine
      # ...
      
  ## TESTED
  app-tested:
    image: mbt-demo:mysql
    # ...
    depends_on:
      - mysql
  mysql:
    image: mysql:8.0
    # ...

Testcontainers

At this point, we could start the application and databases manually for testing, but this would be tedious. Instead, let’s use Testcontainers’ ComposeContainer to automate this with our Docker Compose file during the testing phase.

In this example, we’ll use jqwik as our JUnit 5 test runner. First, let’s add the jqwik and Testcontainers and the jqwik-testcontainers dependencies to our pom.xml:

<dependency>
    <groupId>net.jqwik</groupId>
    <artifactId>jqwik</artifactId>
    <version>1.9.0</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>net.jqwik</groupId>
    <artifactId>jqwik-testcontainers</artifactId>
    <version>0.5.2</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.20.1</version>
    <scope>test</scope>
</dependency>

As a result, we can now instantiate a ComposeContainer and pass our test docker-compose file as argument:

@Testcontainers
class ModelBasedTest {

    @Container
    static ComposeContainer ENV = new ComposeContainer(new File("src/test/resources/docker-compose-test.yml"))
       .withExposedService("app-tested", 8080, Wait.forHttp("/api/employees").forStatusCode(200))
       .withExposedService("app-model", 8080, Wait.forHttp("/api/employees").forStatusCode(200));

    // tests
}

Test HTTP client

Now, let’s create a small test utility that will help us execute the HTTP requests against our services:

class TestHttpClient {
  ApiResponse<EmployeeDto> get(String employeeNo) { /* ... */ }
  
  ApiResponse<Void> put(String employeeNo, String newName) { /* ... */ }
  
  ApiResponse<List<EmployeeDto>> getByDepartment(String department) { /* ... */ }
  
  ApiResponse<EmployeeDto> post(String employeeNo, String name) { /* ... */ }

    
  record ApiResponse<T>(int statusCode, @Nullable T body) { }
    
  record EmployeeDto(String employeeNo, String name) { }
}

Additionally, in the test class, we can declare another method that helps us create TestHttpClients for the two services started by the ComposeContainer:

static TestHttpClient testClient(String service) {
  int port = ENV.getServicePort(service, 8080);
  String url = "http://localhost:%s/api/employees".formatted(port);
  return new TestHttpClient(service, url);
}

jqwik

Jqwik is a property-based testing framework for Java that integrates with JUnit 5, automatically generating test cases to validate properties of code across diverse inputs. By using generators to create varied and random test inputs, jqwik enhances test coverage and uncovers edge cases.

If you’re new to jqwik, you can explore their API in detail by reviewing the official user guide. While this tutorial won’t cover all the specifics of the API, it’s essential to know that jqwik allows us to define a set of actions we want to test.

To begin with, we’ll use jqwik’s @Property annotation — instead of the traditional @Test — to define a test:

@Property
void regressionTest() {
  TestHttpClient model = testClient("app-model");
  TestHttpClient tested = testClient("app-tested");
  // ...
}

Next, we’ll define the actions, which are the HTTP calls to our APIs and can also include assertions.

For instance, the GetOneEmployeeAction will try to fetch a specific employee from both services and compare the responses:

record ModelVsTested(TestHttpClient model, TestHttpClient tested) {}

record GetOneEmployeeAction(String empNo) implements Action<ModelVsTested> {
  @Override
  public ModelVsTested run(ModelVsTested apps) {
    ApiResponse<EmployeeDto> actual = apps.tested.get(empNo);
    ApiResponse<EmployeeDto> expected = apps.model.get(empNo);

    assertThat(actual)
      .satisfies(hasStatusCode(expected.statusCode()))
      .satisfies(hasBody(expected.body()));
    return apps;
  }
}

Additionally, we’ll need to wrap these actions within Arbitrary objects. We can think of Arbitraries as objects implementing the factory design pattern that can generate a wide variety of instances of a type, based on a set of configured rules.

For instance, the Arbitrary returned by employeeNos() can generate employee numbers by choosing a random department from the configured list and concatenating a number between 0 and 200:

static Arbitrary<String> employeeNos() {
  Arbitrary<String> departments = Arbitraries.of("Frontend", "Backend", "HR", "Creative", "DevOps");
  Arbitrary<Long> ids = Arbitraries.longs().between(1, 200);
  return Combinators.combine(departments, ids).as("%s-%s"::formatted);
}

Similarly, getOneEmployeeAction() returns an Aribtrary action based on a given Arbitrary employee number:

static Arbitrary<GetOneEmployeeAction> getOneEmployeeAction() {
  return employeeNos().map(GetOneEmployeeAction::new);
}

After declaring all the other Actions and Arbitraries, we’ll create an ActionSequence:

@Provide
Arbitrary<ActionSequence<ModelVsTested>> mbtJqwikActions() {
  return Arbitraries.sequences(
    Arbitraries.oneOf(
      MbtJqwikActions.getOneEmployeeAction(),
      MbtJqwikActions.getEmployeesByDepartmentAction(),
      MbtJqwikActions.createEmployeeAction(),
      MbtJqwikActions.updateEmployeeNameAction()
  ));
}


static Arbitrary<Action<ModelVsTested>> getOneEmployeeAction() { /* ... */ }
static Arbitrary<Action<ModelVsTested>> getEmployeesByDepartmentAction() { /* ... */ }
// same for the other actions

Now, we can write our test and leverage jqwik to use the provided actions to test various sequences. Let’s create the ModelVsTested tuple and use it to execute the sequence of actions against it:

@Property
void regressionTest(@ForAll("mbtJqwikActions") ActionSequence<ModelVsTested> actions) {
  ModelVsTested testVsModel = new ModelVsTested(
    testClient("app-model"),
    testClient("app-tested")
  );
  actions.run(testVsModel);
}

That’s it — we can finally run the test! The test will generate a sequence of thousands of requests trying to find inconsistencies between the model and the tested service:

INFO com.etr.demo.utils.TestHttpClient -- [app-tested] PUT /api/employeesFrontend-129?name=v
INFO com.etr.demo.utils.TestHttpClient -- [app-model] PUT /api/employeesFrontend-129?name=v
INFO com.etr.demo.utils.TestHttpClient -- [app-tested] GET /api/employees/Frontend-129
INFO com.etr.demo.utils.TestHttpClient -- [app-model] GET /api/employees/Frontend-129
INFO com.etr.demo.utils.TestHttpClient -- [app-tested] POST /api/employees { name=sdxToS, empNo=Frontend-91 }
INFO com.etr.demo.utils.TestHttpClient -- [app-model] POST /api/employees { name=sdxToS, empNo=Frontend-91 }
INFO com.etr.demo.utils.TestHttpClient -- [app-tested] PUT /api/employeesFrontend-4?name=PZbmodNLNwX
INFO com.etr.demo.utils.TestHttpClient -- [app-model] PUT /api/employeesFrontend-4?name=PZbmodNLNwX
INFO com.etr.demo.utils.TestHttpClient -- [app-tested] GET /api/employees/Frontend-4
INFO com.etr.demo.utils.TestHttpClient -- [app-model] GET /api/employees/Frontend-4
INFO com.etr.demo.utils.TestHttpClient -- [app-tested] GET /api/employees?department=ٺ⯟桸
INFO com.etr.demo.utils.TestHttpClient -- [app-model] GET /api/employees?department=ٺ⯟桸
        ...

Catching errors

If we run the test and check the logs, we’ll quickly spot a failure. It appears that when searching for employees by department with the argument ٺ⯟桸 the model produces an internal server error, while the test version returns 200 OK:

Original Sample
---------------
actions:
ActionSequence[FAILED]: 8 actions run [
    UpdateEmployeeAction[empNo=Creative-13, newName=uRhplM],
    CreateEmployeeAction[empNo=Backend-184, name=aGAYQ],
    UpdateEmployeeAction[empNo=Backend-3, newName=aWCxzg],
    UpdateEmployeeAction[empNo=Frontend-93, newName=SrJTVwMvpy],
    UpdateEmployeeAction[empNo=Frontend-129, newName=v],
    CreateEmployeeAction[empNo=Frontend-91, name=sdxToS],
    UpdateEmployeeAction[empNo=Frontend-4, newName=PZbmodNLNwX],
    GetEmployeesByDepartmentAction[department=ٺ⯟桸]
]
    final currentModel: ModelVsTested[model=com.etr.demo.utils.TestHttpClient@5dc0ff7d, tested=com.etr.demo.utils.TestHttpClient@64920dc2]
Multiple Failures (1 failure)
    -- failure 1 --
    expected: 200
    but was: 500

Upon investigation, we find that the issue arises from a native SQL query using Postgres-specific syntax to retrieve data. While this was a simple issue in our small application, model-based testing can help uncover unexpected behavior that may only surface after a specific sequence of repetitive steps pushes the system into a particular state.

Wrap up

In this post, we provided hands-on examples of how model-based testing works in practice. From defining models to generating test cases, we’ve seen a powerful approach to improving test coverage and reducing manual effort. Now that you’ve seen the potential of model-based testing to enhance software quality, it’s time to dive deeper and tailor it to your own projects.

Clone the repository to experiment further, customize the models, and integrate this methodology into your testing strategy. Start building more resilient software today!

Thank you to Emanuel Trandafir for contributing this post.

Learn more

• Clone the model-based testing practice repo.
• Subscribe to the Docker Newsletter. 
• Visit the Testcontainers website.
• Get started with Testcontainers Cloud by creating a free account.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

Cloud Expo Asia 2024 in Singapore drew thousands of cloud professionals and tech business leaders to explore and exchange the latest in cloud computing, security, GenAI, sustainability, DevOps, and more. At our Cloud Expo Asia booth, Docker showcased our latest innovations in AI integration, containerization, security best practices, and updated product offerings. Here are a few highlights from our experience at the event.

AI/ML and GenAI everywhere

AI/ML and GenAI were hot topics at Cloud Expo Asia. Docker CPO Giri Sreenivas’s talk on Transforming App Development: Docker’s Advanced Containerization and AI Integration highlighted that GenAI impacts software in two big ways — it accelerates product development and creates new types of products and experiences. He discussed how containers are an ideal tool for containerizing GenAI workflows in development, ensuring consistency across CI/CD pipelines and reproducibility across diverse platforms in production.

Sreenivas highlighted the Docker extension for GitHub Copilot as an example of how Docker helps empower development teams to focus on innovation — closing the gap from the first line of code to production. Sreenivas also gave a sneak peek into upcoming products designed to streamline GenAI development to illustrate Docker’s commitment to evolving solutions to meet emerging needs. 

Adopting security best practices and shifting left

Developer efficiency and security were also popular themes at the event. When Sreenivas mentioned in his talk that security vulnerabilities that cost dollars to fix early in development would cost hundreds of dollars later in production, members of the audience nodded in agreement.

Docker CTO Justin Cormack gave a keynote address titled “The Docker Effect: Driving Developer Efficiency and Innovation in a Hybrid World.” He discussed how implementing best practices and investing in the inner loop are crucial for today’s development teams. 

One best practice, for example, is shifting left and identifying problems as quickly as possible in the software development lifecycle. This approach improves efficiency and reduces costs by detecting and addressing software issues earlier before they become expensive problems.

Cormack also provided a few tips for meeting the security and control needs of modern enterprises with a layered approach. Start with key building blocks, he explained, such as trusted content, which provides dev teams with a good foundation to build securely from the start. 

At the Docker event booth, we demonstrated Docker Scout, which helps development teams identify, analyze, and remediate security vulnerabilities early in the dev process. Docker Business customers can take advantage of enterprise controls, letting admins, IT teams, and security teams continuously monitor and manage risk and compliance with confidence. 

New Docker innovations and updated plan

From students to C-level executives who visited our booth, everyone was eager to learn more about containers and Docker. People lined up to see an end-to-end demo of how the suite of Docker products, such as Docker Desktop, Docker Hub, Docker Build Cloud, and Docker Scout, work together seamlessly to enable development teams to work more efficiently. 

Attendees also had the opportunity to learn more about Docker’s updated plans, which makes accessing the full suite of Docker products and solutions easy, with options for individual developers, small teams, and large enterprises.

Thanks, Cloud Expo Asia!

We enjoyed our conversations with event attendees and appreciate everyone who helped make this such a successful event. Thank you to the organizers, speakers, sponsors, and the community for a productive, information-packed experience.

From accelerating app development, supporting best practices of shifting left, meeting the security and control needs of modern enterprises, and innovating with GenAI, Docker wants to be your trusted partner to navigate the challenges in modern app development. 

Explore our Docker updated plans to learn how Docker can empower your teams, or contact our sales team to discover how we can help you innovate with confidence.

Learn more

• Find out how we’re investing in innovation and customer relationships.
• Read about updated Docker plans: Simpler, More Value, Better Development and Productivity.
• Subscribe to the Docker Newsletter. 

This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.

At Docker Labs, we’ve been exploring how LLMs can connect different parts of the developer workflow, bridging gaps between tools and processes. A key insight is that LLMs excel at fixing code issues when they have the right context. To provide this context, we’ve developed a process that maps out the codebase using linting violations and the structure of top-level code blocks. 

By combining these elements, we teach the LLM to construct a comprehensive view of the code, enabling it to fix issues more effectively. By leveraging containerization, integrating these tools becomes much simpler.

Previously, my linting process felt a bit disjointed. I’d introduce an error, run Pylint, and receive a message that was sometimes cryptic, forcing me to consult Pylint’s manual to understand the issue. When OpenAI released ChatGPT, the process improved slightly. I could run Pylint, and if I didn’t grasp an error message, I’d copy the code and the violation into GPT to get a better explanation. Sometimes, I’d ask it to fix the code and then manually paste the solution back into my editor.

However, this approach still required several manual steps: copying code, switching between applications, and integrating fixes. How might we improve this process?

Docker’s AI Tools for Devs prompt runner is an architecture that allows us to integrate tools like Pylint directly into the LLM’s workflow through containerization. By containerizing Pylint and creating prompts that the LLM can use to interact with it, we’ve developed a system where the LLM can access the necessary tools and context to help fix code issues effectively.

Understanding the cognitive architecture

For the LLM to assist effectively, it needs a structured way of accessing and processing information. In our setup, the LLM uses the Docker prompt runner to interact with containerized tools and the codebase. The project context is extracted using tools such as Pylint and Tree-sitter that run against the project. This context is then stored and managed, allowing the LLM to access it when needed.

By having access to the codebase, linting tools, and the context of previous prompts, the LLM can understand where problems are, what they are, and have the right code fragments to fix them. This setup replaces the manual process of finding issues and feeding them to the LLM with something automatic and more engaging.

Streamlining the workflow

Now, within my workflow, I can ask the assistant about code quality and violations directly. The assistant, powered by an LLM, has immediate access to a containerized Pylint tool and a database of my code through the Docker prompt runner. This integration allows the LLM to use tools to assist me directly during development, making the programming experience more efficient.

This approach helps us rethink how we interact with our tools. By enabling a conversational interface with tools that map code to issues, we’re exploring possibilities for a more intuitive development experience. Instead of manually finding problems and feeding them to an AI, we can convert our relationship with tools themselves to be conversational partners that can automatically detect issues, understand the context, and provide solutions.

Walking through the prompts

Our project is structured around a series of prompts that guide the LLM through the tasks it needs to perform. These prompts are stored in a Git repository and can be versioned, tracked, and shared. They form the backbone of the project, allowing the LLM to interact with tools and the codebase effectively. We automate this entire process using Docker and a series of prompts stored in a Git repository. Each prompt corresponds to a specific task in the workflow, and Docker containers ensure a consistent environment for running tools and scripts.

Workflow steps

An immediate and existential challenge we encountered was that this class of problem has a lot of opportunities to overwhelm the context of the LLM. Want to read a source code file? It has to be small enough to read. Need to work on more than one file? Your realistic limit is three to four files at once. To solve this, we can instruct the LLM to automate its own workflow with tools, where each step runs in a Docker container.

Again, each step in this workflow runs in a Docker container, which ensures a consistent and isolated environment for running tools and scripts. The first four steps prepare the agent to be able to extract the right context for fixing violations. Once the agent has the necessary context, the LLM can effectively fix the code issues in step 5.

1. Generate violations report using Pylint:

Run Pylint to produce a violation report.

2. Create a SQLite database:

Set up the database schema to store violation data and code snippets.

3. Generate and run INSERT statements:

• Decouple violations from the range they represent.
• Use a script to convert every violation and range from the report into SQL insert statements.
• Run the statements against the database to populate it with the necessary data.

4. Index code in the database:

• Generate an abstract syntax tree (AST) of the project with Tree-sitter (Figure 1).

• Find all second-level nodes (Figure 2). In Python’s grammar, second-level nodes are statements inside of a module.

• Index these top-level ranges into the database.
• Populate a new table to store the source code at these top-level ranges.

5. Fix violations based on context:

Once the agent has gathered and indexed the necessary context, use prompts to instruct the LLM to query the database and fix the code issues (Figure 3).

Each step from 1 to 4 builds the foundation for step 5, where the LLM, with the proper context, can effectively fix violations. The structured preparation ensures that the LLM has all the information it needs to address code issues with precision.

Refining the context for LLM fixes

To understand how our system improves code fixes, let’s consider a specific violation flagged by Pylint. Say we receive a message that there’s a violation on line 60 of our code file block_listed_name.py:

{
  "type": "convention",
  "module": "block_listed_name",
  "obj": "do_front",
  "line": 60,
  "column": 4,
  "endLine": 60,
  "endColumn": 7,
  "path": "cloned_repo/naming_conventions/block_listed_name.py",
  "symbol": "disallowed-name",
  "message": "Disallowed name \"foo\"",
  "message-id": "C0104"
}

From this Pylint violation, we know that the variable foo is a disallowed name. However, if we tried to ask the LLM to fix this issue based solely on this snippet of information, the response wouldn’t be as effective. Why? The LLM lacks the surrounding context — the full picture of the function in which this violation occurs.

This is where indexing the codebase becomes essential

Because we’ve mapped out the codebase, we can now ask the LLM to query the index and retrieve the surrounding code that includes the do_front function. The LLM can even generate the SQL query for us, thanks to its knowledge of the database schema. Once we’ve retrieved the full function definition, the LLM can work with a more complete view of the problem:

def do_front(front_filename, back_filename):
   """
   Front strategy: loop over front image,
   detect blue pixels there,
   substitute in pixels from back.
   Return changed front image.
   """
   foo = SimpleImage(front_filename)
   back = SimpleImage(back_filename)
   for y in range(foo.height):xc
       for x in range(foo.width):
           pixel = foo.get_pixel(x, y)
           # Detect blue pixels in front and replace with back pixels
           if pixel[2] > 2 * max(pixel[0], pixel[1]):
               back_pixel = back.get_pixel(x, y)
               foo.set_pixel(x, y, back_pixel)
   return foo

Now that the LLM can see the whole function, it’s able to propose a more meaningful fix. Here’s what it suggests after querying the indexed codebase and running the fix:

def do_front(front_filename, back_filename):
 """
    Front strategy: loop over front image,
 detect blue pixels there,
    substitute in pixels from back.
    Return changed front image.
    """
    front_image = SimpleImage(front)
    back_image = SimpleImage(back_filename)
    for y in range(front_image.height):
        for x in range(front_image.width            pixel = front_image.get_pixel(x y)
            # Detect blue pixels in front and replace with back pixels
            if pixel[2 >  * max(pixel[0 pixel[1]):
                back_pixel = back_image.get_pixel(x,)
                front_image.set_pixel(x,, back_pixel)
    return front_image

Here, the variable foo has been replaced with the more descriptive front_image, making the code more readable and understandable. The key step was providing the LLM with the correct level of detail — the top-level range — instead of just a single line or violation message. With the right context, the LLM’s ability to fix code becomes much more effective, which ultimately streamlines the development process.

Remember, all of this information is retrieved and indexed by the LLM itself through the prompts we’ve set up. Through this series of prompts, we’ve reached a point where the assistant has a comprehensive understanding of the codebase. 

At this stage, not only can I ask for a fix, but I can even ask questions like “what’s the violation at line 60 in naming_conventions/block_listed_name.py?” and the assistant responds with:

On line 60 of naming_conventions/block_listed_name.py, there's a violation: Disallowed name 'foo'. The variable name 'foo' is discouraged because it doesn't convey meaningful information about its purpose.

Although Pylint has been our focus here, this approach points to a new conversational way to interact with many tools that map code to issues. By integrating LLMs with containerized tools through architectures like the Docker prompt runner, we can enhance various aspects of the development workflow.

We’ve learned that combining tool integration, cognitive preparation of the LLM, and a seamless workflow can significantly improve the development experience. This integration allows an LLM to use tools to directly help while developing, and while Pylint has been the focus here, this also points to a new conversational way to interact with many tools that map code to issues.

To follow along with this effort, check out the GitHub repository for this project.

For more on what we’re doing at Docker, subscribe to our newsletter.

Learn more

• Read the Docker Labs GenAI series.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

We are thrilled to announce that Granite models, IBM’s family of open source and proprietary models built for business, as well as Red Hat InstructLab model alignment tools, are now available on Docker Hub. 

Now, developer teams can easily access, deploy, and scale applications using IBM’s AI models specifically designed for developers.

This news will be officially announced during the AI track of the keynote at IBM TechXchange on October 22. Attendees will get an exclusive look at how IBM’s Granite models on Docker Hub accelerate AI-driven application development across multiple programming languages.

Why Granite on Docker Hub?

With a principled approach to data transparency, model alignment, and security, IBM’s open source Granite models represent a significant leap forward in natural language processing. The models are available under an Apache 2.0 license, empowering developer teams to bring generative AI into mission-critical applications and workflows. 

Granite models deliver superior performance in coding and targeted language tasks at lower latencies, all while requiring a fraction of the compute resources and reducing the cost of inference. This efficiency allows developers to experiment, build, and scale generative AI applications both on-premises and in the cloud, all within departmental budgetary limits.

Here’s what this means for you:

• Simplified deployment: Pull the Granite image from Docker Hub and get up and running in minutes.
• Scalability: Docker offers a lightweight and efficient method for scaling artificial intelligence and machine learning (AI/ML) applications. It allows you to run multiple containers on a single machine or distribute them across different machines in a cluster, enabling horizontal scalability.
• Flexibility: Customize and extend the model to suit your specific needs without worrying about underlying infrastructure.
• Portability: By creating Docker images once and deploying them anywhere, you eliminate compatibility problems and reduce the need for configurations. 
• Community support: Leverage the vast Docker and IBM communities for support, extensions, and collaborations.

In addition to the IBM Granite models, Red Hat also made the InstructLab model alignment tools available on Docker Hub. Developers using InstructLab can adapt pre-trained LLMs using far less real-world data and computing resources than alternative methodologies. InstructLab is model-agnostic and can be used to fine-tune any LLM of your choice by providing additional skills and knowledge.

With IBM Granite AI models and InstructLab available on Docker Hub, Docker and IBM enable easy integration into existing environments and workflows.

Getting started with Granite

You can find the following images available on Docker Hub:

• InstructLab: Ideal for desktop or Mac users looking to explore InstructLab, this image provides a simple introduction to the platform without requiring specialized hardware. It’s perfect for prototyping and testing before scaling up.
• Granite-7b-lab: This image is optimized for model serving and inference on desktop or Mac environments, using the Granite-7B model. It allows for efficient and scalable inference tasks without needing a GPU, perfect for smaller-scale deployments or local testing.

How to pull and run IBM Granite images from Docker Hub 

IBM Granite provides a toolset for building and managing cloud-native applications. Follow these steps to pull and run an IBM Granite image using Docker and the CLI. You can follow similar steps for the Red Hat InstructLab images.

Authenticate to Docker Hub

Enter your Docker username and password when prompted.

Pull the IBM Granite Image

Pull the IBM Granite image from Docker Hub.  

• redhat/granite-7b-lab-gguf: For Mac/desktop users with no GPU support

Run the Image in a Container

Start a container with the IBM Granite image. The container can be started in two modes: CLI (default) and server.

To start the container in CLI mode, run the following:
docker run --ipc=host -it redhat/granite-7b-lab-gguf 

This command opens an interactive bash session within the container, allowing you to use the tools.

To run the container in server mode, run the following command:

docker run --ipc=host -it redhat/granite-7b-lab-gguf -s

You can check IBM Granite’s documentation for details on using IBM Granite Models.

Join us at IBM TechXchange

Granite on Docker Hub will be officially announced at the IBM TechXchange Conference, which will be held October 21-24 in Las Vegas. Our head of technical alliances, Eli Aleyner, will show a live demonstration at the AI track of the keynote during IBM TechXchange. Oleg Šelajev, Docker’s staff developer evangelist, will show how app developers can test their GenAI apps with local models. Additionally, you’ll learn how Docker’s collaboration with Red Hat is improving developer productivity.

The availability of Granite on Docker Hub marks a significant milestone in making advanced AI models accessible to all. We’re excited to see how developer teams will harness the power of Granite to innovate and solve complex challenges.

Stay anchored for more updates, and as always, happy coding!

Learn more

• Read the Docker Labs GenAI series.
• Subscribe to the Docker Newsletter. 
• What is InstructLab?
• What are Granite Models?
• Accelerating AI Development with IBM Granite AI Models and Docker — IBM TechXchange session with Eli Aleyner.
• Developer productivity for apps with AI – IBM TechXchange session with Oleg Šelajev.

We’re excited to announce the launch of the Docker Terraform Provider, designed to help users and organizations automate and securely manage their Docker-hosted resources. This includes repositories, teams, organization settings, and more, all using Terraform’s infrastructure-as-code approach. This provider brings a unified, scalable, and secure solution for managing Docker resources in an automated fashion — whether you’re managing a single repository or a large-scale organization.

A new way of working with Docker Hub

The Docker Terraform Provider introduces a new way of working with Docker Hub, enabling infrastructure-as-code best practices that are already widely adopted across cloud-native environments. By integrating Docker Hub with Terraform, organizations can streamline resource management, improve security, and collaborate more effectively, all while ensuring Docker resources remain in sync with other infrastructure components.

The Problem

Managing Docker Hub resources manually can become cumbersome and prone to errors, especially as teams grow and projects scale. Maintaining configurations can lead to inconsistencies, reduced security, and a lack of collaboration between teams without a streamlined, version-controlled system. The Docker Terraform Provider solves this by allowing you to manage Docker Hub resources in the same way you manage your other cloud resources, ensuring consistency, auditability, and automation across the board.

The solution

The Docker Terraform Provider offers:

• Unified management: With this provider, you can manage Docker repositories, teams, users, and organizations in a consistent workflow, using the same code and structure across environments.
• Version control: Changes to Docker Hub resources are captured in your Terraform configuration, providing a version-controlled, auditable way to manage your Docker infrastructure.
• Collaboration and automation: Teams can now collaborate seamlessly, automating the provisioning and management of Docker Hub resources with Terraform, enhancing productivity and ensuring best practices are followed.
• Scalability: Whether you’re managing a few repositories or an entire organization, this provider scales effortlessly to meet your needs.

Example

At Docker, even we faced challenges managing our Docker Hub resources, especially when adding repositories without owner permissions — it was a frustrating, manual process. With the Terraform provider, anyone in the company can create a new repository without having elevated Docker Hub permissions. All levels of employees are now empowered to write code rather than track down coworkers. This streamlines developer workflows with familiar tooling and reduces employee permissions. Security and developers are happy!

Here’s an example where we are managing a repository, an org team, the permissions for the created repo, and a PAT token:

terraform {
  required_providers {
    docker = {
      source  = "docker/docker"
      version = "~> 0.2"
    }
  }
}

# Initialize provider
provider "docker" {}

# Define local variables for customization
locals {
  namespace        = "my-docker-namespace"
  repo_name        = "my-docker-repo"
  org_name         = "my-docker-org"
  team_name        = "my-team"
  my_team_users    = ["user1", "user2"]
  token_label      = "my-pat-token"
  token_scopes     = ["repo:read", "repo:write"]
  permission       = "admin"
}

# Create repository
resource "docker_hub_repository" "org_hub_repo" {
  namespace        = local.namespace
  name             = local.repo_name
  description      = "This is a generic Docker repository."
  full_description = "Full description for the repository."
}

# Create team
resource "docker_org_team" "team" {
  org_name         = local.org_name
  team_name        = local.team_name
  team_description = "Team description goes here."
}

# Team association
resource "docker_org_team_member" "team_membership" {
  for_each = toset(local.my_team_users)

  org_name  = local.org_name
  team_name = docker_org_team.team.team_name
  user_name = each.value
}

# Create repository team permission
resource "docker_hub_repository_team_permission" "repo_permission" {
  repo_id    = docker_hub_repository.org_hub_repo.id
  team_id    = docker_org_team.team.id
  permission = local.permission
}

# Create access token
resource "docker_access_token" "access_token" {
  token_label = local.token_label
  scopes      = local.token_scopes
}

Future work

We’re just getting started with the Docker Terraform Provider, and there’s much more to come. Future work will expand support to other products in Docker’s suite, including Docker Scout, Docker Build Cloud, and Testcontainers Cloud. Stay tuned as we continue to evolve and enhance the provider with new features and integrations.

For feedback and issue tracking, visit the official Docker Terraform Provider repository or submit feedback via our issue tracker.

We’re confident this new provider will enhance how teams work with Docker Hub, making it easier to manage, secure, and scale their infrastructure while focusing on what matters most — building great software.

Learn more

• Visit the official Docker Terraform Provider repository.
• Submit feedback via our issue tracker.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

If you’ve worked with Docker for any length of time, you’re likely accustomed to writing or at least modifying a Dockerfile. This file can be thought of as a recipe for a Docker image; it contains both the ingredients (base images, packages, files) and the instructions (various RUN, COPY, and other commands that help build the image).

In most cases, Dockerfiles are written once, modified seldom, and used as-is unless something about the project changes. Because these files are created or modified on such an infrequent basis, developers tend to rely on only a handful of frequently used instructions — RUN, COPY, and EXPOSE being the most common. Other instructions can enhance your image, making it more configurable, manageable, and easier to maintain. 

In this post, we will discuss the ARG and ENV instructions and explore why, how, and when to use them.

ARG: Defining build-time variables

The ARG instruction allows you to define variables that will be accessible during the build stage but not available after the image is built. For example, we will use this Dockerfile to build an image where we make the variable specified by the ARG instruction available during the build process.

FROM ubuntu:latest
ARG THEARG="foo"
RUN echo $THEARG
CMD ["env"]

If we run the build, we will see the echo foo line in the output:

$ docker build --no-cache -t argtest .
[+] Building 0.4s (6/6) FINISHED                                                                     docker:desktop-linux
<-- SNIP -->
 => CACHED [1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => [2/2] RUN echo foo                                                                                               0.1s
 => exporting to image                                                                                               0.0s
<-- SNIP -->

However, if we run the image and inspect the output of the env command, we do not see THEARG:

$ docker run --rm argtest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=d19f59677dcd
HOME=/root

ENV: Defining build and runtime variables

Unlike ARG, the ENV command allows you to define a variable that can be accessed both at build time and run time:

FROM ubuntu:latest
ENV THEENV="bar"
RUN echo $THEENV
CMD ["env"]

If we run the build, we will see the echo bar line in the output:

$ docker build -t envtest .
[+] Building 0.8s (7/7) FINISHED                                                                     docker:desktop-linux
<-- SNIP -->
 => CACHED [1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => [2/2] RUN echo bar                                                                                               0.1s
 => exporting to image                                                                                               0.0s
<-- SNIP -->

If we run the image and inspect the output of the env command, we do see THEENV set, as expected:

$ docker run --rm envtest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=f53f1d9712a9
THEENV=bar
HOME=/root

Overriding ARG

A more advanced use of the ARG instruction is to serve as a placeholder that is then updated at build time:

FROM ubuntu:latest
ARG THEARG
RUN echo $THEARG
CMD ["env"]

If we build the image, we see that we are missing a value for $THEARG:

$ docker build -t argtest .
<-- SNIP -->
 => CACHED [1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => [2/2] RUN echo $THEARG                                                                                           0.1s
 => exporting to image                                                                                               0.0s
 => => exporting layers                                                                                              0.0s
<-- SNIP -->

However, we can pass a value for THEARG on the build command line using the --build-arg argument. Notice that we now see THEARG has been replaced with foo in the output:

 => CACHED [1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => [2/2] RUN echo foo                                                                                               0.1s
 => exporting to image                                                                                               0.0s
 => => exporting layers                                                                                              0.0s
<-- SNIP -->

The same can be done in a Docker Compose file by using the args key under the build key. Note that these can be set as a mapping (THEARG: foo) or a list (- THEARG=foo):

services:
  argtest:
    build:
      context: .
      args:
        THEARG: foo

If we run docker compose up --build, we can see the THEARG has been replaced with foo in the output:

$ docker compose up --build
<-- SNIP -->
 => [argtest 1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => CACHED [argtest 2/2] RUN echo foo                                                                                0.0s
 => [argtest] exporting to image                                                                                     0.0s
 => => exporting layers                                                                                              0.0s
<-- SNIP -->
Attaching to argtest-1
argtest-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
argtest-1  | HOSTNAME=d9a3789ac47a
argtest-1  | HOME=/root
argtest-1 exited with code 0

Overriding ENV

You can also override ENV at build time; this is slightly different from how ARG is overridden. For example, you cannot supply a key without a value with the ENV instruction, as shown in the following example Dockerfile:

FROM ubuntu:latest
ENV THEENV
RUN echo $THEENV
CMD ["env"]

When we try to build the image, we receive an error:

$ docker build -t envtest .
[+] Building 0.0s (1/1) FINISHED                                                                     docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                 0.0s
 => => transferring dockerfile: 98B                                                                                  0.0s
Dockerfile:3
--------------------
   1 |     FROM ubuntu:latest
   2 |
   3 | >>> ENV THEENV
   4 |     RUN echo $THEENV
   5 |
--------------------
ERROR: failed to solve: ENV must have two arguments

However, we can remove the ENV instruction from the Dockerfile:

FROM ubuntu:latest
RUN echo $THEENV
CMD ["env"]

This allows us to build the image:

$ docker build -t envtest .
<-- SNIP -->
 => [1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => CACHED [2/2] RUN echo $THEENV                                                                                    0.0s
 => exporting to image                                                                                               0.0s
 => => exporting layers                                                                                              0.0s
<-- SNIP -->

Then we can pass an environment variable via the docker run command using the -e flag:

$ docker run --rm -e THEENV=bar envtest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=638cf682d61f
THEENV=bar
HOME=/root

Although the .env file is usually associated with Docker Compose, it can also be used with docker run.

$ cat .env
THEENV=bar

$ docker run --rm --env-file ./.env envtest
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=59efe1003811
THEENV=bar
HOME=/root

This can also be done using Docker Compose by using the environment key. Note that we use the variable format for the value:

services:
  envtest:
    build:
      context: .
    environment:
      THEENV: ${THEENV}

If we do not supply a value for THEENV, a warning is thrown:

$ docker compose up --build
WARN[0000] The "THEENV" variable is not set. Defaulting to a blank string.
<-- SNIP -->
 => [envtest 1/2] FROM docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04  0.0s
 => => resolve docker.io/library/ubuntu:latest@sha256:8a37d68f4f73ebf3d4efafbcf66379bf3728902a8038616808f04e34a9ab6  0.0s
 => CACHED [envtest 2/2] RUN echo ${THEENV}                                                                          0.0s
 => [envtest] exporting to image                                                                                     0.0s
<-- SNIP -->
 ✔ Container dd-envtest-1    Recreated                                                                               0.1s
Attaching to envtest-1
envtest-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
envtest-1  | HOSTNAME=816d164dc067
envtest-1  | THEENV=
envtest-1  | HOME=/root
envtest-1 exited with code 0

The value for our variable can be supplied in several different ways, as follows:

• On the compose command line:

$ THEENV=bar docker compose up

[+] Running 2/0
 ✔ Synchronized File Shares                                                                                          0.0s
 ✔ Container dd-envtest-1    Recreated                                                                               0.1s
Attaching to envtest-1
envtest-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
envtest-1  | HOSTNAME=20f67bb40c6a
envtest-1  | THEENV=bar
envtest-1  | HOME=/root
envtest-1 exited with code 0

• In the shell environment on the host system:

$ export THEENV=bar
$ docker compose up

[+] Running 2/0
 ✔ Synchronized File Shares                                                                                          0.0s
 ✔ Container dd-envtest-1    Created                                                                                 0.0s
Attaching to envtest-1
envtest-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
envtest-1  | HOSTNAME=20f67bb40c6a
envtest-1  | THEENV=bar
envtest-1  | HOME=/root
envtest-1 exited with code 0

• In the special .env file:

$ cat .env
THEENV=bar

$ docker compose up

[+] Running 2/0
 ✔ Synchronized File Shares                                                                                          0.0s
 ✔ Container dd-envtest-1    Created                                                                                 0.0s
Attaching to envtest-1
envtest-1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
envtest-1  | HOSTNAME=20f67bb40c6a
envtest-1  | THEENV=bar
envtest-1  | HOME=/root
envtest-1 exited with code 0

Finally, when running services directly using docker compose run, you can use the -e flag to override the .env file.

$ docker compose run -e THEENV=bar envtest

[+] Creating 1/0
 ✔ Synchronized File Shares                                                                                          0.0s
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=219e96494ddd
TERM=xterm
THEENV=bar
HOME=/root

The tl;dr

If you need to access a variable during the build process but not at runtime, use ARG. If you need to access the variable both during the build and at runtime, or only at runtime, use ENV.

To decide between them, consider the following flow (Figure 1):

Both ARG and ENV can be overridden from the command line in docker run and docker compose, giving you a powerful way to dynamically update variables and build flexible workflows.

Learn more

• Discover more Docker Best Practices.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

At Docker, innovation and efficiency are integral to how we operate. When our own IT team needed to deploy Docker Desktop to various teams, including non-engineering roles like customer support and technical sales, the existing process was functional but manual and time-consuming. Recognizing the need for a more streamlined and secure approach, we leveraged new Early Access (EA) Docker Business features to refine our deployment strategy.

A seamless deployment process

Faced with the challenge of managing diverse requirements across the organization, we knew it was time to enhance our deployment methods.

The Docker IT team transitioned from using registry.json files to a more efficient method involving registry keys and new MSI installers for Windows, along with configuration profiles and PKG installers for macOS. This transition simplified deployment, provided better control for admins, and allowed for faster rollouts across the organization.

“From setup to deployment, it took 24 hours. We started on a Monday morning, and by the next day, it was done,” explains Jeffrey Strauss, Head of Docker IT. 

Enhancing security and visibility

Security is always a priority. By integrating login enforcement with single sign-on (SSO) and System for Cross-domain Identity Management (SCIM), Docker IT ensured centralized control and compliance with security policies. The Docker Desktop Insights Dashboard (EA) offered crucial visibility into how Docker Desktop was being used across the organization. Admins could now see which versions were installed and monitor container usage, enabling informed decisions about updates, resource allocation, and compliance. (Docker Business customers can learn more about access and timelines by contacting their account reps. The Insights Dashboard is only available to Docker Business customers with enforced authentication for organization users.)

Steven Novick, Docker’s Principal Product Manager, emphasized, “With the new solution, deployment was simpler and tamper-proof, giving a clear picture of Docker usage within the organization.”

Benefits beyond deployment

The improvements made by Docker IT extended beyond just deployment efficiency:

• Improved visibility: The Insights Dashboard provided detailed data on Docker usage, helping ensure all users are connected to the organization.
• Efficient deployment: Docker Desktop was deployed to hundreds of computers within 24 hours, significantly reducing administrative overhead.
• Enhanced security: Centralized control to enforce authentication via MDM tools like Intune for Windows and Jamf for macOS strengthened security and compliance.
• Seamless user experience: Early and transparent communication ensured a smooth transition, minimizing disruptions.

Looking ahead

The successful deployment of Docker Desktop within 24 hours demonstrates Docker’s commitment to continuous improvement and innovation. We are excited about the future developments in Docker Desktop management and look forward to supporting our customers as they achieve their goals with Docker. 

Existing Docker Business customers can learn more about access and timelines by contacting their account reps. The Insights Dashboard is only available in Early Access to select Docker Business customers with enforced authentication for organization users.

Curious about how Docker’s new features can benefit your team? Get in touch to discover more or explore our customer stories to see how others are succeeding with Docker.

Learn more

• Read the Docker IT case study to learn more.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

In the past, securely managing access to organization resources has been difficult. The only way to gain access has been through an assigned user’s personal access tokens. Whether these users are your engineer’s accounts, bot accounts, or service accounts, they often become points of risk for your organization.

Now, we’re pleased to introduce a long-awaited feature: organization access tokens.

Organization access tokens are like personal access tokens, but at an organizational level with many improvements and features. In this post, we walk through a few reasons why this feature release is so exciting.

Frictionless management

Every day, we are reducing the friction for organizations and engineers using our products. We want you working on your projects, not managing your development tools. 

Organization access tokens do not require you to manage groups and repository assignments like users require. This means you benefit from a straightforward way to manage access that each access token has instead of managing users and their placement within the organization.

If your organization has SSO enabled and enforced, you have likely run into the issue where machine or service accounts cannot log in easily because they don’t have the ability to log into your identity provider. With organization access tokens, this is no longer a problem.

Did someone leave your organization? No problem! With organization access tokens, you are still in control of the token instead of having to track down which tokens were on that user’s account and deal with the resulting challenges.

Fine-grained access

Organization access tokens introduce a new way to allow for tokens to access resources within your organization. These tokens can be assigned to specific repositories with specific actions for full access management with “least privilege” applied. Of course, you can also allow access to all resources in your organization.

Expirations

Another critical feature is the ability to set expirations for your organization access tokens. This is great for customers who have compliance requirements for token rotation or for those who just like the extra security.

Visibility

Management and registry actions all show up in your organization’s activity logs for each access token. Each token’s usage also shows up on your organization’s usage reports.

Business use cases and fair use

We believe that organization access tokens are useful in the context of teams and companies, which is why we are making them available to Docker Team and Docker Business subscribers. With the usual attention to the security aspect, avoiding any “misuse” related to the proliferation of the number of access tokens created, we are introducing a limitation in the maximum number of organization access tokens based on the type of subscription. There will be a limit of 10 for Team plans and 100 for Business plans.

Try organization access tokens

If you are on a team or business subscription, check out our documentation to learn more about using organization access tokens.

Learn more

• Read the organization access tokens documentation.
• Learn about Docker subscription plans.
• Subscribe to the Docker Newsletter.
• Authenticate and update to receive your subscription level’s newest Docker features.
• New to Docker? Create an account.

DevOps brings together developers and operations teams to create better software by introducing organizational principles that encourage communication, collaboration, innovation, speed, security, and agility throughout the software development lifecycle. And, the popularity and adoption rates of DevOps continue to grow, with 83% of 10,000 global developers surveyed saying that they use the principles, according to an April 2024 report commissioned by the Continuous Delivery Foundation (CDF), a Linux Foundation project.

DevOps includes everything from continuous integration/improvement and continuous deployment/delivery (CI/CD) as code is created and modified, to critical automation capabilities covering a wide range of development processes. Also built into DevOps principles is a focus on creating better applications from code conception all the way through to end-user experiences. Before this unified framework existed, code typically was created in separate silos that did not easily allow collaboration or foster efficient management, speed, or quality. These conditions eventually inspired the DevOps framework and principles.  

DevOps principles and practices also help organizations by constantly integrating user feedback regarding application features, shortcomings, and code glitches, thereby reducing security and operational risks in code as it reaches production.

This blog post aims to help enterprises focus on one of these critical DevOps capabilities in particular — the use of automation to speed and streamline processes across the development lifecycle of applications — to further expand and drive the benefits of using DevOps processes within an organization.

As DevOps use continues to grow, more developers are finding that the Docker containerization platform integrates well as a crucial component of DevOps practices, especially due to its built-in automation features and capabilities.

What is DevOps automation?

DevOps automation is a major time-saver for developers and operations teams because it automates labor-intensive and repetitive processes that can free up developers to instead work on new code innovations and ideas that can create business value.  

Automating repetitive manual tasks using DevOps automation tools drives notable efficiencies and productivity boosts for developers and organizations, using automatic actions that eliminate frequent developer or operations team intervention. 

What DevOps processes can you automate?

DevOps automation is especially valuable because it can be used on a broad spectrum of tasks in the application development environment, including CI/CD pipelines and workflows, code writing, monitoring and logging, and Infrastructure as Code (IaC) tools. It can also help improve and streamline configuration management, infrastructure provisioning, unit tests, code testing, security steps and scans, troubleshooting, code review, deploying and delivering code, project management, and more.

By bringing beneficial and time-saving automation to the DevOps lifecycle, developers can create cleaner and more secure code with much less manual intervention and human error compared to traditional software development methods. 

Benefits of DevOps automation tools

For development and operations teams, using DevOps automation to streamline and improve their operations goes far beyond just reducing human error rates and increasing the efficiency and speed of code creation and the deployment process.

Other benefits of DevOps automation include improved consistency and reliability, delivery of predictable and repeatable results, and enhanced scalability and manageability of multiple applications and processes. These benefits become possible with automation because it reduces many human mistakes and miscalculations.

DevOps automation benefits can also include smoother collaboration among multiple developers working on applications at the same time by automatically handling merge conflicts, and performing automatic code testing for multiple developers at once. Automation that troubleshoots applications can also speed up project development times by immediately notifying systems personnel of problems as they arise.

How to automate DevOps with Docker

As a flexible tool for DevOps automation, Docker is available in four subscription levels, from the free Docker Personal version to the top-of-the-line Docker Business tier. 

Docker Business delivers a wide range of helpful tools that empower DevOps teams to identify development bottlenecks where automation can free up resources and resolve repetitive tasks and operations. The following tools are included with Docker Business. (Read our September 2024 announcement about upgraded Docker subscription plans that will deliver even more value, flexibility, and power to your development workflows.) 

Docker Image Access Management

With Docker Business, developers and operations teams can quickly start automating tasks using features such as Docker Image Access Management, which gives administrators control over the types of container images that developers can pull and use from Docker Hub. This includes Docker Official Images, Docker Verified Publisher Images, and community images. Using Image Access Management, developers and teams can more easily search private registries and community repositories for needed container images to use to build their applications. 

Image Access Management allows organizations to give developers freedom of choice while providing some guardrails to prevent developers from accidentally using untrusted, malicious community images as components of their applications. This is an important benefit, compared with only allowing developers to use a handful of internally built images, for example.

Docker Image Access Management is available only to Docker Business customers.  

Docker automated testing 

Other Docker DevOps automation features include automated testing, including source code repository testing, that can be done through Docker Hub to automatically test changes to source code repositories using containers. Any Docker Hub repository can enable an autotest function to run tests on pull requests to the source code repository to create a continuous integration testing service.

Automated test files to perform the tests can be set up by creating a docker-compose.test.yml file, which defines a service that lists the tests to be run. The docker-compose.test.yml file should be placed in the same directory that contains the Dockerfile used to build the image.

Hardened Docker Desktop

To automate security within Docker, administrators can use a wide range of features within Hardened Docker Desktop, which is available to Docker Business subscribers. Hardened Docker Desktop security features aim to bolster the security of developer environments while causing minimal speed or performance impacts on developer experiences or productivity. 

These features allow administrators to enforce strict security settings, which prevent developers and containers from bypassing the controls intentionally or unintentionally. The features also enable enhanced container isolation capabilities to prevent potential security threats, such as malicious payloads, from breaching the Docker Desktop Linux VM and the underlying host.

Using Hardened Docker Desktop, security administrators can take more control and ownership over Docker Desktop configurations, removing and preventing potential changes by users, which is vital for security-conscious organizations.

Automated builds

Another automation and productivity tool is the Docker Automated builds feature, which automatically builds images from source code in an external repository and then pushes the built image to designated Docker repositories. Available in the Docker Business, Pro, or Teams tiers, Automated builds — also called autobuilds — create a list of branches and tags that can be built into Docker images using a series of commands. Automated builds can handle images of up to 10 GB in size.

Enhanced collaboration tools 

Throughout Docker’s unified suite, tools built to deliver enhanced collaboration are available to developers and operations teams to work together to get the most out of their projects and applications.

Everything from Docker Desktop to Docker Engine, Docker CLI, Docker Compose, Docker Build/BuildKit, Docker Desktop Extensions, and more are designed to enable developers and operations teams to accelerate productivity, reduce code errors, increase security, drive innovation, and save valuable time throughout the software development process. 

Easier scaling and orchestration with Kubernetes integration

Docker’s containerization platform also integrates well with the Kubernetes container orchestration platform, optimizing the developer experience for container development, deployment, and management. Docker and Kubernetes can work together using Docker Engine as a user-friendly and secure foundation for basic Kubernetes (K8s) functionality, or by using Docker Desktop for a more comprehensive approach that avoids potential challenges associated with do-it-yourself container configurations. Docker Desktop includes K8s setup at the push of a button, which is one of its numerous and useful automation features. 

Support and troubleshooting 

As Docker continues to mature, its knowledge base is constantly being expanded and deepened, with core documentation and resources freely available to Docker developers within the Docker ecosystem. And, because Docker uses a collaborative approach between developers and operations teams, developers can often find common answers to their inquiries and learn from each other to tackle most issues.

More information and help about using Docker can be found in the Docker Training page, which offers live and on-demand training and other resources to help developers and teams negotiate their Docker landscapes and learn fresh skills to resolve technical problems. 

Other resources: Docker Scout and Docker Build Cloud

Docker offers even more tools to help with automation, collaboration, and creating better and more nimble code for developer teams and operations managers.

Docker Scout, for example, is built to help organizations better protect their software supply chain security when using container images, which may contain software elements that are susceptible to security vulnerabilities. 

Docker Scout helps with this issue by proactively analyzing container images and compiling a Software Bill of Materials (SBOM), which is a detailed inventory of code included in an application or container. That SBOM is then matched against a continuously updated vulnerability database to pinpoint and correct security weaknesses to help make the code more secure.

Docker Build Cloud is a Docker service to help developers build container images more quickly, both locally and in the cloud. Those builds run on cloud infrastructure that requires no configuration and where the environment is optimally dimensioned for all workloads using a remote build cache. This approach ensures fast builds anywhere for all team members. 

To use Docker Build Cloud, developers take the same steps they would take for a regular build using the command docker buildx build. With a regular build command, the build runs on a local instance of BuildKit, bundled with the Docker daemon. But when using Docker Build Cloud, the build request is sent to a BuildKit instance running remotely, in the cloud, with all data encrypted in transit. Docker Build Cloud provides several benefits over local builds, including faster build speed, shared build cache, and native multi-platform builds.

Future trends in DevOps automation

As DevOps automation continues to mature, it will gain more capabilities from artificial intelligence (AI), machine learning (ML), serverless architectures, cloud-native platforms, and other technologies across the IT landscape. 

Such advancements can be found in Docker’s AI collaborations with NVIDIA. For example, Docker Desktop dovetails with the NVIDIA AI Workbench, which is an easy-to-use toolkit that lets developers create, test, and customize AI and machine learning models on a PC or workstation and then scale them to a data center or public cloud. NVIDIA AI Workbench makes interactive development workflows easier, while automating technical tasks that can halt beginners and derail experts. 

DevOps automation is ripe for further improvements and enhancements from AI and ML in areas of agility, process improvements, and more for developers and operations teams. AI and ML will drive further labor savings for software development teams by delivering fresh new automated, self-service tools that free them up from a broader range of routine tasks, giving them more time to conduct valuable and critical work that will drive their companies forward.

Docker will be an important part of this changing landscape as the unified suites and tools continue to expand and deliver further new benefits and capabilities to DevOps, the Docker ecosystem, and developers and operations teams around the world.

Wrapping up

Improving DevOps automation by using the Docker containerization platform inside your business organization is a smart strategy that helps developers and operations teams deliver their best work with efficiency, creativity, and broad collaboration.

Docker Business plays a leadership role in enhancing DevOps automation in companies around the world as they look to automate their DevOps operations effectively.

Ready to automate your team’s DevOps processes? Find out how Docker Business can transform your development, or if you still have questions, reach out to one of our experts to get started!

Learn more

• Subscribe to the Docker Newsletter. 
• Learn about the 2024 announcement of upgraded Docker plans that are simpler and offer even more value. 
• Learn how other DevOps teams use Docker. 
• Find the perfect pricing for your team. 
• Try the latest version of Docker Desktop. 
• Visit Docker Resources to explore more materials.

I recently joined Docker in January as Chief Revenue Officer. My role is responsible for the entire customer journey, from your first interaction with Docker’s sales org to post-sales support and onboarding. As I speak with customers and hear stories about their journey with Docker over the past decade, I’m often reminded of the immense trust you’ve placed in us. Whether you’ve been with us from the days of Docker Swarm or have more recently started using Docker Desktop, your partnership has been invaluable in shaping who we are today. 

I want to take a moment to personally thank you for being part of our story, especially as we continue to evolve in a rapidly changing ecosystem.

We know that change can bring challenges. Over the years, as containers became the backbone of modern software development, Docker has evolved alongside them. This evolution has not always been easy and I understand that shifts in our product offerings, changes in pricing, and recent adjustments to our subscription plans have impacted many of you. Our priority now, as it always has been, is to deliver unrivaled value to you. 

We recognize that to continue innovating and addressing the complex needs of modern developers, we must continue to invest in Docker products and our relationships with customers like you. This investment isn’t just about tools and features; it’s about creating a holistic ecosystem — a unified suite — that makes your development process more productive, secure, and manageable at an enterprise scale, while building a go-to-market organization that is equipped to support our growing customer base. 

1. To that end, we’ve redefined our strategy to focus on a deeper, more meaningful engagement with you. We’re committed to building stronger relationships, listening carefully to your feedback, and ensuring that the solutions we bring to market truly address your pain points. By focusing on your needs, we’re working to make every interaction with Docker more valuable, whether it’s through enhanced support, new features, or better licensing management. If you’d like to discuss this with me further, I’m happy to schedule time. (Reach out by email or connect with your Account Executive to set this up.)

2. Additionally, we’ve made key investments in our enterprise suite of products that surrounds Docker Desktop. We understand that the demands of modern development extend beyond the individual developer’s experience. Docker is the only container-first platform built specifically for development teams, improving developer experience and productivity while meeting the security and control needs of modern enterprises. Docker offers a comprehensive suite of enterprise-ready tools, cloud services, trusted content, and a collaborative community that helps streamline workflows and maximize development efficiency.

As we continue to invest in both vectors above, we’re excited about what lies ahead in our product roadmap. Our aim is simple: to help your teams develop with confidence, knowing that Docker is a trusted partner invested in your success. I am personally dedicated to ensuring that our roadmap reflects your needs and that our solutions empower your teams to reach their full potential.

Thank you again for your continued trust and partnership. We wouldn’t be here without you, and I look forward to what we will achieve together.

Learn more

• Read Announcing Upgraded Docker Plans: Simpler, More Value, Better Development and Productivity.
• Subscribe to the Docker Newsletter. 

This post was contributed by Jesús Espino, Principal Engineer at Mattermost.

In the ever-evolving software development landscape, ensuring robust and reliable plugin integration is no small feat. For Mattermost, relying solely on mocks for plugin testing became a limitation, leading to brittle tests and overlooked integration issues. Enter Testcontainers, an open source tool that provides isolated Docker environments, making complex integration testing not only feasible but efficient. 

In this blog post, we dive into how Mattermost has embraced Testcontainers to overhaul its testing strategy, achieving greater automation, improved accuracy, and seamless plugin integration with minimal overhead.

The previous approach

In the past, Mattermost relied heavily on mocks to test plugins. While this approach had its merits, it also had significant drawbacks. The tests were brittle, meaning they would often break when changes were made to the codebase. This made the tests challenging to develop and maintain, as developers had to constantly update the mocks to reflect the changes in the code.

Furthermore, the use of mocks meant that the integration aspect of testing was largely overlooked. The tests did not account for how the different components of the system interacted with each other, which could lead to unforeseen issues in the production environment. 

The previous approach additionally did not allow for proper integration testing in an automated way. The lack of automation made the testing process time-consuming and prone to human error. These challenges necessitated a shift in Mattermost’s testing strategy, leading to the adoption of Testcontainers for complex integration testing.

Mattermost’s approach to integration testing

Testcontainers for Go

Mattermost uses Testcontainers for Go to create an isolated testing environment for our plugins. This testing environment includes the Mattermost server, the PostgreSQL server, and, in certain cases, an API mock server. The plugin is then installed on the Mattermost server, and through regular API calls or end-to-end testing frameworks like Playwright, we perform the required testing.

We have created a specialized Testcontainers module for the Mattermost server. This module uses PostgreSQL as a dependency, ensuring that the testing environment closely mirrors the production environment. Our module allows the developer to install and configure any plugin you want in the Mattermost server easily.

To improve the system’s isolation, the Mattermost module includes a container for the server and a container for the PostgreSQL database, which are connected through an internal Docker network.

Additionally, the Mattermost module exposes utility functionality that allows direct access to the database, to the Mattermost API through the Go client, and some utility functions that enable admins to create users, channels, teams, and change the configuration, among other things. This functionality is invaluable for performing complex operations during testing, including API calls, users/teams/channel creation, configuration changes, or even SQL query execution. 

This approach provides a powerful set of tools with which to set up our tests and prepare everything for verifying the behavior that we expect. Combined with the disposable nature of the test container instances, this makes the system easy to understand while remaining isolated.

This comprehensive approach to testing ensures that all aspects of the Mattermost server and its plugins are thoroughly tested, thereby increasing their reliability and functionality. But, let’s see a code example of the usage.

We can start setting up our Mattermost environment with a plugin like this:

pluginConfig := map[string]any{}
options := []mmcontainer.MattermostCustomizeRequestOption{
  mmcontainer.WithPlugin("sample.tar.gz", "sample", pluginConfig),
}
mattermost, err := mmcontainer.RunContainer(context.Background(), options...)
defer mattermost.Terminate(context.Background()

Once your Mattermost instance is initialized, you can create a test like this:

func TestSample(t *testing.T) {
    client, err mattermost.GetClient()
    require.NoError(t, err)
    reqURL := client.URL + "/plugins/sample/sample-endpoint"
    resp, err := client.DoAPIRequest(context.Background(), http.MethodGet, reqURL, "", "")
    require.NoError(t, err, "cannot fetch url %s", reqURL)
    defer resp.Body.Close()
    bodyBytes, err := io.ReadAll(resp.Body)
    require.NoError(t, err)
    require.Equal(t, 200, resp.StatusCode)
    assert.Contains(t, string(bodyBytes), "sample-response") 
}

Here, you can decide when you tear down your Mattermost instance and recreate it. Once per test? Once per a set of tests? It is up to you and depends strictly on your needs and the nature of your tests.

Testcontainers for Node.js

In addition to using Testcontainers for Go, Mattermost leverages Testcontainers for Node.js to set up our testing environment. In case you’re unfamiliar, Testcontainers for Node.js is a Node.js library that provides similar functionality to Testcontainers for Go. Using Testcontainers for Node.js, we can set up our environment in the same way we did with Testcontainers for Go. This allows us to write Playwright tests using JavaScript and run them in the isolated Mattermost environment created by Testcontainers, enabling us to perform integration testing that interacts directly with the plugin user interface. The code is available on GitHub.  

This approach provides the same advantages as Testcontainers for Go, and it allows us to use a more interface-based testing tool — like Playwright in this case. Let me show a bit of code with the Node.js and Playwright implementation:

We start and stop the containers for each test:

test.beforeAll(async () => { mattermost = await RunContainer() })
test.afterAll(async () => { await mattermost.stop(); })

Then we can use our Mattermost instance like any other server running to run our Playwright tests:

test.describe('sample slash command', () => {
  test('try to run a sample slash command', async ({ page }) => {
    const url = mattermost.url()
    await login(page, url, "regularuser", "regularuser")
    await expect(page.getByLabel('town square public channel')).toBeVisible();
    await page.getByTestId('post_textbox').fill("/sample run")
    await page.getByTestId('SendMessageButton').click();
    await expect(page.getByText('Sample command result', { exact: true })).toBeVisible();
    await logout(page)
  });  
});

With these two approaches, we can create integration tests covering the API and the interface without having to mock or use any other synthetic environment. Also, we can test things in absolute isolation because we consciously decide whether we want to reuse the Testcontainers instances. We can also reach a high degree of isolation and thereby avoid the flakiness induced by contaminated environments when doing end-to-end testing.

Examples of usage

Currently, we are using this approach for two plugins.

1. Mattermost AI Copilot

This integration helps users in their daily tasks using AI large language models (LLMs), providing things like thread and meeting summarization and context-based interrogation.

This plugin has a rich interface, so we used the Testcontainers for Node and Playwright approach to ensure we could properly test the system through the interface. Also, this plugin needs to call the AI LLM through an API. To avoid that resource-heavy task, we use an API mock, another container that simulates any API.

This approach gives us confidence in the server-side code but in the interface side as well, because we can ensure that we aren’t breaking anything during the development.

2. Mattermost MS Teams plugin

This integration is designed to connect MS Teams and Mattermost in a seamless way, synchronizing messages between both platforms.

For this plugin, we mainly need to do API calls, so we used Testcontainers for Go and directly hit the API using a client written in Go. In this case, again, our plugin depends on a third-party service: the Microsoft Graph API from Microsoft. For that, we also use an API mock, enabling us to test the whole plugin without depending on the third-party service.

We still have some integration tests with the real Teams API using the same Testcontainers infrastructure to ensure that we are properly handling the Microsoft Graph calls.

Benefits of using Testcontainers libraries

Using Testcontainers for integration testing offers benefits, such as:

• Isolation: Each test runs in its own Docker container, which means that tests are completely isolated from each other. This approach prevents tests from interfering with one another and ensures that each test starts with a clean slate.
• Repeatability: Because the testing environment is set up automatically, the tests are highly repeatable. This means that developers can run the tests multiple times and get the same results, which increases the reliability of the tests.
• Ease of use: Testcontainers is easy to use, as it handles all the complexities of setting up and tearing down Docker containers. This allows developers to focus on writing tests rather than managing the testing environment.
  

Testing made easy with Testcontainers

Mattermost’s use of Testcontainers libraries for complex integration testing in their plugins is a testament to the power and versatility of Testcontainers.

By creating a well-isolated and repeatable testing environment, Mattermost ensures that our plugins are thoroughly tested and highly reliable.

Learn more

• Subscribe to the Docker Newsletter. 
• Visit the Testcontainers website.
• Get started with Testcontainers Cloud by creating a free account.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.

LLMs are now quite good at transforming data. For example, we were recently working with some data generated by the Pylint tool. This tool generates big arrays of code violations. 

Here’s an example showing the kind of data that gets returned.

[
    {
        "type": "convention",
        "module": "app",
        "line": 1,
        "column": 0,
        "endLine": 1,
        "endColumn": 13,
        "path": "src/app.py",
        "symbol": "missing-module-docstring",
        "message": "Missing module docstring",
        "message-id": "C0114"
    },
    {
       ...
    },
    ...
]

During this session with our AI assistant, we decided that it would be helpful to create a database and insert the data to make it easier for the AI to analyze (LLMs are very good at writing SQL). As is now our habit, we wrote a quick prompt to see if the assistant could generate the SQL:

1. Read the json data from /thread/violations.json
2. For each element in the array, transform each element into two SQL INSERT statements.

* The first should insert columns PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN into a table named RANGES.
* The second should insert columns MESSAGE, TYPE, RANGE, and VIOLATION_ID into a table named VIOLATIONS.

LLMs are obviously good at tasks of this kind, and this was no exception. Our prompt engine had been augmented with a function to read local files but, besides that, this was a pretty straightforward prompt (we used GPT-3.5). The LLM responded with the correct INSERT statements.

INSERT INTO RANGES (PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN)
VALUES ('src/app.py', 1, 1, 0, 13);
INSERT INTO VIOLATIONS (MESSAGE, TYPE, RANGE, VIOLATION_ID)
VALUES ('Missing module docstring', 'convention', (SELECT ID FROM RANGES WHERE PATH='src/app.py' AND START_LINE=1 AND START_COLUMN=0), 'C0114');
...
...

We’re starting to get accustomed to this kind of capability, so the result wasn’t too surprising. However, what about the context window here? It really doesn’t make a lot of sense to pass all of this data to the LLM, especially if this is a task that we’ll need to do continuously. Plus, this is really not how a programmer would have solved the problem.

Programmers would write programs.

So, instead of asking the LLM to do a thing, we should try asking it to write a program to do that same thing. Starting with the same prompt, let’s prefix it with “Please write a JavaScript program to …” In other words, let’s ask the LLM to describe how it would accomplish the task in JavaScript, and hopefully automate itself out of the loop. Here’s the adjusted prompt:

Write a javascript function that reads json from a file named `/thread/violations.json`.

For each element in the array, transform each element into two SQL INSERT statements.

* The first should insert columns PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN into a table named RANGES.
* The second should insert columns MESSAGE, TYPE, RANGE, and VIOLATION_ID into a table named VIOLATIONS.

and write them to the file /thread/insert.sql

Now execute the javascript code in a container.

Tool code

Current LLMs can do tasks of this kind. Here’s a sample of the output generated by GPT-3.5. A quick scan will convince many of you that this is probably going to do the trick. However, if our prompt engine is already running in Docker, we might as well just run it:

const fs = require('fs');
const crypto = require('crypto');

function getRandomId() {
    return crypto.randomBytes(16).toString('hex');
}

function escapeString(str) {
    return str.replace(/'/g, \\\"''\\\");
}

let data = JSON.parse(fs.readFileSync('/thread/violations.json', 'utf8'));
let sqlStatements = '';

data.forEach((violation) => {
    let id = getRandomId();
    sqlStatements += `INSERT INTO RANGES(ID, PATH, START_LINE, END_LINE, START_COLUMN, END_COLUMN) VALUES('${id}', '${escapeString(violation.path)}', ${violation.line}, ${violation.endLine}, ${violation.column}, ${violation.endColumn});\\\\n`;
    sqlStatements += `INSERT INTO VIOLATIONS(MESSAGE, TYPE, RANGE) VALUES('${escapeString(violation.message)}', '${escapeString(violation.type)}', '${id}');\\\\n`;
    });

fs.writeFileSync('/thread/insert.sql', sqlStatements);

Running the tool

You may not have noticed but at the end of our prompt, we added a final instruction: “Now execute the JavaScript code in a container”. This is a nice addition to our session, because it means we get to see the results.

This is also where tool calling comes back into the picture. To give our AI the capacity to try running the program that it has just written, we have defined a new function to create an isolated runtime sandbox for trying out our new tool.

Here’s the agent’s new tool definition:

tools:
  - name: run-javascript-sandbox
    description: execute javascript code in a container
    parameters:
      type: object
      properties:
        javascript:
          type: string
          description: the javascript code to run
    container:
      image: vonwig/javascript-runner
      command:
        - "{{javascript|safe}}"

We’ve asked the AI assistant to generate a tool from a description of that tool. As long as the description of the tools doesn’t change, the workflow won’t have to go back to the AI to ask it to build a new tool version.

The role of Docker in this pattern is to create the sandbox for this code to run. This function really doesn’t need much of a runtime, so we give it a pretty small sandbox.

• No access to a network.
• No access to the host file system (does have access to isolated volumes for sharing data between tools).
• No access to GPU.
• Almost no access to software besides the Node.js runtime (no shell for example).

The ability for one tool to create another tool is not just a trick. It has very practical implications for the kinds of workflows that we can build up because it gives us a way for us to control the volume of data sent to LLMs, and it gives the assistant a way to “automate” itself out of the loop.

Next steps

This example was a bit abstract but in our next post, we will describe the practical scenarios that have driven us to look at this idea of prompts generating new tools. Most of the workflows we’re exploring are still just off-the-shelf tools like Pylint, SQLite, and tree_sitter (which we embed using Docker, of course!). For example:

1. Use pylint to extract violations from my codebase.
2. Transform the violations into SQL and then send that to a new SQLite.
3. Find the most common violations of type error and show me the top level code blocks containing them.

However, you’ll also see that part of being able to author workflows of this kind is being able to recognize when you just need to add a custom tool to the mix.

Read the Docker Labs GenAI series to see more of what we’ve been working on.

Learn more

• Read the Docker Labs GenAI series.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

With many organizations moving to container-based workflows, keeping track of the different versions of your images can become a problem. Even smaller organizations can have hundreds of container images spanning from one-off development tests, through emergency variants to fix problems, all the way to core production images. This leads us to the question: How can we tame our image sprawl while still rapidly iterating our images?

A common misconception is that by using the “latest” tag, you are guaranteeing that you are pulling the “latest” version of the image. Unfortunately, this assumption is wrong — all latest means is “the last image pushed to this registry.”

Read on to learn more about how to avoid this pitfall when using Docker and how to get a handle on your Docker images.

Using tags

One way to address this issue is to use tags when creating an image. Adding one or more tags to an image helps you remember what it is intended for and helps others as well. One approach is always to tag images with their semantic versioning (semver), which lets you know what version you are deploying. This sounds like a great approach, and, to some extent, it is, but there is a wrinkle.

Unless you’ve configured your registry for immutable tags, tags can be changed. For example, you could tag my-great-app as v1.0.0 and push the image to the registry. However, nothing stops your colleague from pushing their updated version of the app with tag v1.0.0 as well. Now that tag points to their image, not yours. If you add in the convenience tag latest, things get a bit more murky.

Let’s look at an example:

FROM busybox:stable-glibc

# Create a script that outputs the version
RUN echo -e "#!/bin/sh\n" > /test.sh && \
    echo "echo \"This is version 1.0.0\"" >> /test.sh && \
    chmod +x /test.sh

# Set the entrypoint to run the script
ENTRYPOINT ["/bin/sh", "/test.sh"]

We build the above with docker build -t tagexample:1.0.0 . and run it.

$ docker run --rm tagexample:1.0.0
This is version 1.0.0

What if we run it without a tag specified?

$ docker run --rm tagexample
Unable to find image 'tagexample:latest' locally
docker: Error response from daemon: pull access denied for tagexample, repository does not exist or may require 'docker login'.
See 'docker run --help'.

Now we build with docker build . without specifying a tag and run it.

$ docker run --rm tagexample
This is version 1.0.0

The latest tag is always applied to the most recent push that did not specify a tag. So, in our first test, we had one image in the repository with a tag of 1.0.0, but because we did not have any pushes without a tag, the latest tag did not point to an image. However, once we push an image without a tag, the latest tag is automatically applied to it.

Although it is tempting to always pull the latest tag, it’s rarely a good idea. The logical assumption — that this points to the most recent version of the image — is flawed. For example, another developer can update the application to version 1.0.1, build it with the tag 1.0.1, and push it. This results in the following:

$ docker run --rm tagexample:1.0.1
This is version 1.0.1

$ docker run --rm tagexample:latest
This is version 1.0.0

If you made the assumption that latest pointed to the highest version, you’d now be running an out-of-date version of the image.

The other issue is that there is no mechanism in place to prevent someone from inadvertently pushing with the wrong tag. For example, we could create another update to our code bringing it up to 1.0.2. We update the code, build the image, and push it — but we forget to change the tag to reflect the new version. Although it’s a small oversight, this action results in the following:

$ docker run --rm tagexample:1.0.1
This is version 1.0.2

Unfortunately, this happens all too frequently.

Using labels

Because we can’t trust tags, how should we ensure that we are able to identify our images? This is where the concept of adding metadata to our images becomes important.

The first attempt at using metadata to help manage images was the MAINTAINER instruction. This instruction sets the “Author” field (org.opencontainers.image.authors) in the generated image. However, this instruction has been deprecated in favor of the more powerful LABEL instruction. Unlike MAINTAINER, the LABEL instruction allows you to set arbitrary key/value pairs that can then be read with docker inspect as well as other tooling.

Unlike with tags, labels become part of the image, and when implemented properly, can provide a much better way to determine the version of an image. To return to our example above, let’s see how the use of a label would have made a difference.

To do this, we add the LABEL instruction to the Dockerfile, along with the key version and value 1.0.2.

FROM busybox:stable-glibc

LABEL version="1.0.2"

# Create a script that outputs the version
RUN echo -e "#!/bin/sh\n" > /test.sh && \
    echo "echo \"This is version 1.0.2\"" >> /test.sh && \
    chmod +x /test.sh

# Set the entrypoint to run the script
ENTRYPOINT ["/bin/sh", "/test.sh"]

Now, even if we make the same mistake above where we mistakenly tag the image as version 1.0.1, we have a way to check that does not involve running the container to see which version we are using.

$ docker inspect --format='{{json .Config.Labels}}' tagexample:1.0.1
{"version":"1.0.2"}

Best practices

Although you can use any key/value as a LABEL, there are recommendations. The OCI provides a set of suggested labels within the org.opencontainers.image namespace, as shown in the following table:

Because LABEL takes any key/value, it is also possible to create custom labels. For example, labels specific to a team within a company could use the com.myorg.myteam namespace. Isolating these to a specific namespace ensures that they can easily be related back to the team that created the label.

Final thoughts

Image sprawl is a real problem for organizations, and, if not addressed, it can lead to confusion, rework, and potential production problems. By using tags and labels in a consistent manner, it is possible to eliminate these issues and provide a well-documented set of images that make work easier and not harder.

Learn more

• Read the Docker Best Practices collection.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.

DevOps aims to dramatically improve the software development lifecycle by bringing together the formerly separated worlds of development and operations using principles that strive to make software creation more efficient. DevOps practices form a useful roadmap to help developers in every phase of the development lifecycle, from code planning to building, task automation, testing, monitoring, releasing, and deploying applications.

As DevOps use continues to expand, many developers and organizations find that the Docker containerization platform integrates well as a crucial component of DevOps practices. Using Docker, developers have the advantage of being able to collaborate in standardized environments using local containers and remote container tools where they can write their code, share their work, and collaborate. 

In this blog post, we will explore the use of Docker within DevOps practices and explain how the combination can help developers create more efficient and powerful workflows.

What is DevOps?

DevOps practices are beneficial in the world of developers and code creation because they encourage smart planning, collaboration, and orderly processes and management throughout the software development pipeline. Without unified DevOps principles, code is typically created in individual silos that can hamper creativity, efficient management, speed, and quality.

Bringing software developers, operations teams, and processes together under DevOps principles, can improve both developer and organizational efficiency through increased collaboration, agility, and innovation. DevOps brings these positive changes to organizations by constantly integrating user feedback regarding application features, shortcomings, and code glitches and — by making changes as needed on the fly — reducing operational and security risks in production code.

CI/CD

In addition to collaboration, DevOps principles are built around procedures for continuous integration/improvement (CI) and continuous deployment/delivery (CD) of code, shortening the cycle between development and production. This CI/CD approach lets teams more quickly adapt to feedback and thus build better applications from code conception all the way through to end-user experiences.

Using CI, developers can frequently and automatically integrate their changes into the source code as they create new code, while the CD side tests and delivers those vetted changes to the production environment. By integrating CI/CD practices, developers can create cleaner and safer code and resolve bugs ahead of production through automation, collaboration, and strong QA pipelines. 

What is Docker?

The Docker containerization platform is a suite of tools, standards, and services that enable DevOps practices for application developers. Docker is used to develop, ship, and run applications within lightweight containers. This approach allows developers to separate their applications from their business infrastructure, giving them the power to deliver better code more quickly. 

The Docker platform enables developers to package and run their application code in lightweight, local, standardized containers, which provide a loosely isolated environment that contains everything needed to run the application — including tools, packages, and libraries. By using Docker containers on a Docker client, developers can run an application without worrying about what is installed on the host, giving them huge flexibility, security, and collaborative advantages over virtual machines. 

In this controlled environment, developers can use Docker to create, monitor, and push their applications into a test environment, run automated and manual tests as needed, correct bugs, and then validate the code before deploying it for use in production. 

Docker also allows developers to run many containers simultaneously on a host, while allowing those same containers to be shared with others. Such a collaborative workspace can foster healthy and direct communications between developers, allowing development processes to become easier, more accurate, and more secure. 

Containers vs. virtualization

Containers are an abstraction that packages application code and dependencies together. Instances of the container can then be created, started, stopped, moved, or deleted using the Docker API or command-line interface (CLI). Containers can be connected to one or more networks, be attached to storage, or create new images based on their current states. 

Containers differ from virtual machines, which use a software abstraction layer on top of computer hardware, allowing the hardware to be shared more efficiently in multiple instances that will run individual applications. Docker containers require fewer physical hardware resources than virtual machines, and they also offer faster startup times and lower overhead. This makes Docker ideal for high-velocity environments, where rapid software development cycles and scalability are crucial. 

Basic components of Docker 

The basic components of Docker include:

• Docker images: Docker images are the blueprints for your containers. They are read-only templates that contain the instructions for creating a Docker container. You can think of a container image as a snapshot of a specific state of your application.
• Containers: Containers are the instances of Docker images. They are lightweight and portable, encapsulating your application along with its dependencies. Containers can be created, started, stopped, moved, and deleted using simple Docker commands.
• Dockerfiles: A Dockerfile is a text document containing a series of instructions on how to build a Docker image. It includes commands for specifying the base image, copying files, installing dependencies, and setting up the environment. 
• Docker Engine: Docker Engine is the core component of Docker. It’s a client-server application that includes a server with a long-running daemon process, APIs for interacting with the daemon, and a CLI client.
• Docker Desktop: Docker Desktop is a commercial product sold and supported by Docker, Inc. It includes the Docker Engine and other open source components, proprietary components, and features like an intuitive GUI, synchronized file shares, access to cloud resources, debugging features, native host integration, governance, security features, and administrative settings management. 
• Docker Hub: Docker Hub is a public registry where you can store and share Docker images. It serves as a central place to find official Docker images and user-contributed images. You can also use Docker Hub to automate your workflows by connecting it to your CI/CD pipelines.

Basic Docker commands

Docker commands are simple and intuitive. For example:

• docker run: Runs a Docker container from a specified image. For example, docker run hello-world will run a container from the “hello-world” image.
• docker build: Builds an image from a Dockerfile. For example, docker build -t my-app . will build an image named “my-app” from the Dockerfile in the current directory.
• docker pull: Pulls an image from Docker Hub. For example, docker pull nginx will download the latest NGINX image from Docker Hub.
• docker ps: Lists all running containers. For example, docker ps -a will list all containers, including stopped ones.
• docker stop: Stops a running Docker container. For example, docker stop <container_id> will stop the container with the specified ID.
• docker rm: Removes a stopped container. For example, docker rm <container_id> will remove the container with the specified ID.

How Docker is used in DevOps

One of Docker’s most important benefits for developers is its critical role in facilitating CI/CD in the application development process. This makes it easier and more seamless for developers to work together to create better code.

Docker is a build environment where developers can get predictable results building and testing their applications inside Docker containers and where it is easier to get consistent, reproducible results compared to other development environments. Developers can use Dockerfiles to define the exact requirements needed for their build environments, including programming runtimes, operating systems, binaries, and more.

Using Docker as a build environment also makes application maintenance easier. For example, you can update to a new version of a programming runtime by just changing a tag or digest in a Dockerfile. That is easier than the process required on a virtual machine to manually reinstall a newer version and update the related configuration files.

Automated testing is also easier using Docker Hub, which can automatically test changes to source code repositories using containers or push applications into a test environment and run automated and manual tests.

Docker can be integrated with DevOps tools including Jenkins, GitLab, Kubernetes, and others, simplifying DevOps processes by automating pipelines and scaling operations as needed. 

Benefits of using Docker for DevOps 

Because the Docker containers used for development are the same ones that are moved along for testing and production, the Docker platform provides consistency across environments and delivers big benefits to developer teams and operations managers. Each Docker container is isolated from others being run, eliminating conflicting dependencies. Developers are empowered to build, run, and test their code while collaborating with others and using all the resources available to them within the Docker platform environment. 

Other benefits to developers include speed and agility, resource efficiency, error reduction, integrated version control, standardization, and the ability to write code once and run it on any system. Additionally, applications built on Docker can be pushed easily to customers on any computing environment, assuring quick, easy, and consistent delivery and deployment process. 

4 Common Docker challenges in DevOps

Implementing Docker in a DevOps environment can offer numerous benefits, but it also presents several challenges that teams must navigate:

1. Learning curve and skills gap

Docker introduces new concepts and technologies that require teams to acquire new skills. This can be a significant hurdle, especially if the team lacks experience with containerization. Docker’s robust documentation and guides and our international community can help new users quickly ramp up.

2. Security concerns

Ensuring the security of containerized applications involves addressing vulnerabilities in container images, managing secrets, and implementing network policies. Misconfigurations and running containers with root privileges can lead to security risks. Docker does, however, provide security guardrails for both administrators and developers.

The Docker Business subscription provides security and management at scale. For example, administrators can enforce sign-ins across Docker products for developers and efficiently manage, scale, and secure Docker Desktop instances using DevOps security controls like Enhanced Container Isolation and Registry Access Management.

Additionally, Docker offers security-focused tools, like Docker Scout, which helps administrators and developers secure the software supply chain by proactively monitoring image vulnerabilities and implementing remediation strategies. Introduced in 2024, Docker Scout health scores rate the security and compliance status of container images within Docker Hub, providing a single, quantifiable metric to represent the “health” of an image. This feature addresses one of the key friction points in developer-led software security — the lack of security expertise — and makes it easier for developers to turn critical insights from tools into actionable steps.

3. Microservice architectures

Containers and the ecosystem around them are specifically geared towards microservice architectures. You can run a monolith in a container, but you will not be able to leverage all of the benefits and paradigms of containers in that way. Instead, containers can be a useful gateway to microservices. Users can start pulling out individual pieces from a monolith into more containers over time.

4. Image management

Image management in Docker can also be a challenge for developers and teams as they search private registries and community repositories for images to use in building their applications. Docker Image Access Management can help with this challenge as it gives administrators control over which types of images — such as Docker Official Images, Docker Verified Publisher Images, or community images — their developers can pull for use from Docker Hub. Docker Hub tries to help by publishing only official images and verifying content from trusted partners. 

Using Image Access Management controls helps prevent developers from accidentally using an untrusted, malicious community image as a component of their application. Note that Docker Image Access Management is available only to customers of the company’s top Docker Business services offering.

Another important tool here is Docker Scout. It is built to help organizations better protect their software supply chain security when using container images, which consist of layers and software packages that may be susceptible to security vulnerabilities. Docker Scout helps with this issue by proactively analyzing container images and compiling a Software Bill of Materials (SBOM), which is a detailed inventory of code included in an application or container. That SBOM is then matched against a continuously updated vulnerability database to pinpoint and correct security weaknesses to make the code more secure.

More information and help about using Docker can be found in the Docker Trainings page, which offers training webcasts and other resources to assist developers and teams to negotiate their Docker landscapes and learn fresh skills to solve their technical inquiries. 

Examples of DevOps using Docker

Improving DevOps workflows is a major goal for many enterprises as they struggle to improve operations and developer productivity and to produce cleaner, more secure, and better code.

The Warehouse Group

At The Warehouse Group, New Zealand’s largest retail store chain with some 300 stores, Docker was introduced in 2016 to revamp its systems and processes after previous VMware deployments resulted in long setup times, inconsistent environments, and slow deployment cycles. 

“One of the key benefits we have seen from using Docker is that it enables a very flexible work environment,” said Matt Law, the chapter lead of DevOps for the company. “Developers can build and test applications locally on their own machines with consistency across environments, thanks to Docker’s containerization approach.”

Docker brought new autonomy to the company’s developers so they could test ideas and find new and better ways to solve bottlenecks, said Law. “That is a key philosophy that we have here — enabling developers to experiment with tooling to help them prove or disprove their philosophies or theories.”

Ataccama Corporation

Another Docker customer, Ataccama Corp., a Toronto-based data management software vendor, adopted Docker and DevOps practices when it moved to scale its business by moving from physical servers to cloud platforms like AWS and Azure to gain agility, scalability, and cost efficiencies using containerization. 

For Ataccama, Docker delivered rapid deployment, simplified application management, and seamless portability between environments, which brought accelerated feature development, increased efficiency and performance, valuable microservices capabilities, and required security and high availability. To boost the value of Docker for its developers and IT managers, Ataccama provided container and DevOps skills training and promoted collaboration to make Docker an integral tool and platform for the company and its operations.

“What makes Docker a class apart is its support for open standards like Open Container Initiative (OCI) and its amazing flexibility,” said Vladimir Mikhalev, senior DevOps engineer at Ataccama. “It goes far beyond just running containers. With Docker, we can build, share, and manage containerized apps seamlessly across infrastructure in a way that most tools can’t match.”

The most impactful feature of Docker is its ability to bundle an app, configuration, and dependencies into a single standardized unit, said Mikhalev. “This level of encapsulation has been a game-changer for eliminating environment inconsistencies.”

Wrapping up

Docker provides a transformative impact for enterprises that have adopted DevOps practices. The Docker platform enables developers to create, collaborate, test, monitor, ship, and run applications within lightweight containers, giving them the power to deliver better code more quickly. 

Docker simplifies and empowers development processes, enhancing productivity and improving the reliability of applications across different environments. 

Find the right Docker subscription to bolster your DevOps workflow. 

Learn more

• Read Docker for Web Developers: Getting Started with the Basics.
• Subscribe to the Docker Newsletter. 
• Learn how your team can succeed with Docker. 
• Find the perfect pricing for your team. 
• Download the latest version of Docker Desktop. 
• Visit Docker Resources to explore more materials.

Welcome to the third annual Docker State of Application Development survey!

Please help us better understand and serve the application development  community with just 20-30 minutes of your time. We want to know where you’re focused, what you’re working on, and what is most important to you. Your thoughts and feedback will help us build the best products and experiences for you.

And, we don’t just keep this information for ourselves — we share with you¹! We hope you saw our recent report on the 2023 State of Application Development Survey. The engagement of our community allowed us to better understand where developers are facing challenges, what tools they like, and what they’re excited about. We’ve been using this information to give our community the tools and features they need.

Take the Docker State of Application Development survey now!

By participating in the survey, you can be entered into a raffle for a chance to win² one of the following prizes:

• 1 laptop computer (an Apple M3 Macbook Pro 16″)
• 2 game consoles with VR headsets: Playstation Virtual Reality Headset and a Playstation 5
• 5 $300 Amazon.com gift cards 
• 50 exclusive Docker swag sets 

Additionally, the first 200 respondents to complete the survey will receive an exclusive pair of Docker socks!

The survey is open from September 23rd, 2024 (7AM PST) to November 20, 2024 (11:59PM PST). 

We’ll choose the winners randomly in accordance with the promotion official rules.* Winners will be notified via email by January 10, 2025.

The Docker State of Application Development Survey only takes about 20-30 minutes to complete. We appreciate every contribution and opinion. Your voice counts!

1. Data will be reported publicly only in aggregate and without personally identifying information. ↩︎
2. Docker State of Application Development Promotion Official Rules. ↩︎

This ongoing Docker Labs GenAI series explores the exciting space of AI developer tools. At Docker, we believe there is a vast scope to explore, openly and without the hype. We will share our explorations and collaborate with the developer community in real-time. Although developers have adopted autocomplete tooling like GitHub Copilot and use chat, there is significant potential for AI tools to assist with more specific tasks and interfaces throughout the entire software lifecycle. Therefore, our exploration will be broad. We will be releasing software as open source so you can play, explore, and hack with us, too.

Using new tools on the command line can be frustrating. Even if we are confident that we’ve found the right tool, we might not know how to use it.

Telling an agent to RT(F)M

A typical workflow might look something like the following.

• Install tool.
• Read the documentation.
• Run the command.
• Repeat.

Can we improve this flow using LLMs?

Install tool

Docker provides us with isolated environments to run tools. Instead of requiring that commands be installed, we have created minimal Docker images for each tool so that using the tool does not impact the host system. Leave no trace, so to speak.

Read the documentation

Man pages are one of the ways that authors of tools ship content about how to use that tool. This content also comes with standard retrieval mechanisms (the man tool). A tool might also support a command-line option like --help. Let’s start with the idealistic notion that we should be able to retrieve usage information from the tool itself.

In this experiment, we’ve created two entry points for each tool. The first entry point is the obvious one. It is a set of arguments passed directly to a command-line program. The OpenAI-compatible description that we generate for this entry point is shown below. We are using the same interface for every tool.

{"name": "run_my_tool",
   "description": "Run the my_tool command.",
   "parameters":
   {"type": "object",
    "properties":
    {"args":
     {"type": "string",
      "description": "The arguments to pass to my_tool"}}},
   "container": {"image": "namespace/my_tool:latest"}}

The second entrypoint gives the agent the ability to read the man page and, hopefully, improve its ability to run the first entrypoint. The second entrypoint is simpler, because it only does one thing (asks a tool how to use it).

{"name": "my_tool_manual",
   "description": "Read the man page for my_tool",
   "container": {"image": "namespace/my_tool:latest", "command": ["man"]}}

Run the command

Let’s start with a simple example. We want to use a tool called qrencode to generate a QR code for a link. We have used our image generation pipeline to package this tool into a minimal image for qrencode. We will now pass this prompt to a few different LLMs; we are using LLMs that have been trained for tool calling (e.g., GPT 4, Llama 3.1, and Mistral). Here’s the prompt that we are testing:

Generate a QR code for the content https://github.com/docker/labs-ai-tools-for-devs/blob/main/prompts/qrencode/README.md. Save the generated image to qrcode.png.
If the command fails, read the man page and try again.

Note the optimism in this prompt. Because it’s hard to predict what different LLMs have already seen in their training sets, and many command-line tools use common names for arguments, it’s interesting to see what LLM will infer before adding the man page to the context.

The output of the prompt is shown below. Grab your phone and check it out.

Repeat

When an LLM generates a description of how to run something, it will usually format that output in such a way that it will be easy for a user to cut and paste the response into a terminal:

qrencode -o qrcode.png 'my content'

However, if the LLM is generating tool calls, we’ll see output that is instead formatted to be easier to run:

[{"function": {"arguments": "{
  \"args\": \"-o qrcode.png 'my content'\"
}"
               "name": "qrencode"}
  "id": "call_Vdw2gDFMAwaEUMgxLYBTg8MB"}]

We respond to this by spinning up a Docker container.

Running the tool as part of the conversation loop is useful even when the command fails. In Unix, there are standard ways to communicate failures. For example, we have exit codes, and stderr streams. This is how tools create feedback loops and correct our behavior while we’re iterating at the terminal. This same mechanism can be used in a conversation loop involving an LLM.

To illustrate, here is another simple example. We’ll try running a tool that writes ASCII art with the following prompt.

Write ascii art with the message "Docker" using the font "helvetica".  Read the man page for figlet if you detect any errors and then try again.

In our test, this did fail. However, it also described the apparent issue on the stderr output stream.

Error: error: could not load font helvetica

By including this message in the conversation loop, the assistant can suggest different courses of action. Different LLMs produced different results here. For example, Llama 3.1 gives instructions for how to install the missing font. On the other hand, GPT 4 re-ran the command, but only after having made the “executive” decision to try a different font.

I'm sorry, it seems the font Helvetica is not available. Let's try it with the default font.

Here is the output of the figlet command with the default font (term): 

 ____             _             
|   \    _       | | _____ _ 
| | | |/  \ / _| |/ /  \ '_|
| |_| | (_) | (__|   <  __/ |   
|____/ \___/ \___|_|\_\___|_|

We are very early in understanding how to take advantage of this apparent capacity to try different approaches. But this is another reason why quarantining these tools in Docker containers is useful. It limits their blast radius while we encourage experimentation.

Results

We started by creating a pipeline to produce minimal Docker images for each tool. The set of tools was selected based on whether they have outputs useful for developer-facing workflows. We continue to add new tools as we think of new use cases. The initial set is listed below.

gh pylint commitlint scalafix gitlint yamllint checkmake gqlint sqlint golint golangci-lint hadolint markdownlint-cli2 cargo-toml-lint ruff dockle clj-kondo selene tflint rslint yapf puppet-lint oxlint kube-linter csslint cpplint ansible-lint actionlint black checkov jfmt datefmt rustfmt cbfmt yamlfmt whatstyle rufo fnlfmt shfmt zprint jet typos docker-ls nerdctl diffoci dive kompose git-test kubectl fastly infracost sops curl fzf ffmpeg babl unzip jq graphviz pstree figlet toilet tldr qrencode clippy go-tools ripgrep awscli2 azure-cli luaformatter nixpkgs-lint hclfmt fop dnstracer undocker dockfmt fixup_yarn_lock github-runner swiftformat swiftlint nix-linter go-critic regal textlint formatjson5 commitmsgfmt

There was a set of initial problems with context extraction.

Missing manual pages

Only about 60% of the tools we selected have man pages. However, even in those cases, there are usually other ways to get help content. The following steps show the final procedure we used:

• Try to run the man page.
• Try to run the tool with the argument --help.
• Try to run the tool with the argument -h.
• Try to run the tool with --broken args and then read stderr.

Using this procedure, every tool in the list above eventually produced documentation.

Long manual pages

Limited context lengths impacted some of the longer manual pages, so it was still necessary to employ standard RAG techniques to summarize verbose man pages. Our tactic was to focus on descriptions of command-line arguments and sections that had sample usage. These had the largest impact on the quality of the agent’s output. The structure of Unix man pages helped with the chunking, because we were able to rely on standard sections to chunk the content.

Subcommands

For a small set of tools, it was necessary to traverse a tree of help menus. However, these were all relatively popular tools, and the LLMs we deployed already knew about this command structure. It’s easy to check this out for yourself. Ask an LLM, for example: “What are the subcommands of Git?” or “What are the subcommands of Docker?” Maybe only popular tools get big enough that they start to be broken up into subcommands.

Summary

We should consider the active role that agents can play when determining how to use a tool. The Unix model has given us standards such as man pages, stderr streams, and exit codes, and we can take advantage of these conventions when asking an assistant to learn a tool. Beyond distribution, Docker also provides us with process isolation, which is useful when creating environments for safe exploration.

Whether or not an AI can successfully generate tool calls may also become a metric for whether or not a tool has been well documented.

To follow along with this effort, check out the GitHub repository for this project.

Learn more

• Read the Docker Labs GenAI series.
• Subscribe to the Docker Newsletter. 
• Get the latest release of Docker Desktop.
• Vote on what’s next! Check out our public roadmap.
• Have questions? The Docker community is here to help.
• New to Docker? Get started.