Deploying Custom ML Models on the DJI Matrice 4E

Deploy custom ML models on DJI drones for real-time object detection. Complete tutorial: training, optimization, and deployment using DJI's AI platform.

Deploying Custom ML Models on the DJI Matrice 4E

Introduction

The integration of machine learning capabilities with drone technology has opened up a world of possibilities for various industries. From infrastructure inspections and wildlife conservation to search and rescue operations, the ability to deploy custom machine learning models directly on drones represents a significant advancement in aerial robotics. In a recent Drone Software meetup, Godfrey Nolan, President at RIIS, showed off just how easy this has become with the DJI Matrice 4E with DJI’s developer tools. This article explores how to train and deploy custom object detection models on DJI’s industrial Matrice 4E drone using YOLOv8, offering a streamlined approach that eliminates the need to build custom applications from scratch. As usual, you can follow along with the meetup video or the written tutorial below.

Evolution of ML on Drones

The DJI Matrice 4E provides a powerful platform for deploying machine learning models, allowing users to focus on developing accurate detection models rather than recreating complex drone control applications. This approach represents a significant departure from previous methods, where custom apps had to be built using the DJI SDK. Now, with the ability to upload models directly to the drone, users can leverage the familiar DJI Pilot 2 interface while adding custom object detection capabilities.

Machine learning on drones is not a new concept. RIIS had worked on applications like cattle counting from drones for about five to six years, dating back to the DJI Phantom 4 era. These early applications required building custom mobile applications that incorporated both the DJI SDK for drone control and TensorFlow for object detection, but now with the DJI Matrice 4E (M4E) we no longer need to create a custom app.

The Process

DJI has made it relatively easy to deploy custom models on the M4E. Here are the steps:

Apply to become an Algorithm Dev via the DJI Developer Portal
Train their models using DJI recommended structure
1. We’re using MMYOLO an open-source toolbox and framework for YOLO series algorithms, built on top of PyTorch
Submit the model to DJI for quantization and optimization
1. Includes .pth file + calibration images
DJI quantizes and evaluates the model
Upload the processed model directly to the drone
Access detection capabilities through the standard DJI Pilot 2 interface

This workflow dramatically reduces development costs and time while improving the user experience. End users can continue working with the familiar DJI Pilot interface rather than learning a new application. Additionally, the processing happens on the drone itself rather than on the remote controller or mobile device.

Okay, now that we know the steps, let’s go into a little more detail for each of the steps. and then later, we’ll dive into the nuances of training the model.

Getting Access to DJI’s AI Platform

The first step in the process is to apply for access to DJI’s AI development platform. Unlike the mobile SDK, which simply requires an API key, the AI platform requires explicit permission from DJI. According to Nolan, DJI states that the approval process typically takes 8-10 days.

To apply for access, developers need to visit the DJI Developer Portal and follow the application process. This is a crucial first step and should be initiated well in advance of when you plan to deploy your model.

Dataset Preparation

One of the most challenging aspects of training object detection models is obtaining and labeling suitable training data. For drone applications, this typically involves flying the drone to collect footage and then manually labeling objects of interest in the images. For our example, we’re using a Kaggle dataset for Potholes, Cracks, and Manholes to make it easier to get started.

The DJI Matrice 4E currently supports detecting up to ten different object classes in a single model. Modern models typically require 500-1,000 labeled images for reasonable performance, significantly fewer than the 5,000+ images needed in earlier approaches.

Model Training Environment Setup

Training a custom object detection model for the DJI Matrice 4E requires a suitable development environment. The recommended setup includes:

Linux operating system (although Windows with WSL would also work)
NVIDIA GPU for reasonable training times
Anaconda/Miniconda for environment management
MMYOLO, a PyTorch wrapper that simplifies the training process

Model Training with MMYOLO

MMYOLO serves as a wrapper for PyTorch and YOLOv8, making the training process more streamlined and efficient. The framework allows for easy swapping between different YOLO versions and accelerates the training process compared to working directly with PyTorch.

Setting Up the Training Environment

The initial setup involves installing Anaconda/Miniconda and MMYOLO. DJI provides a patch file that needs to be applied to make the models compatible with their system:

First, download and install Miniconda.

After installing Miniconda, MMYOLO needs to be set up with DJI’s specific patch:

Clone the repository with git clone <https://github.com/open-mmlab/mmyolo.git>
Navigate to the directory and check out the correct version: cd mmyolo followed by git checkout tags/v0.6.0
Create a new branch with git switch -c drone-model-training
Apply DJI’s patch file with git apply ~/Downloads/ai_inside_model/0001-NEW-ai-inside-init.patch

Once MMYOLO is installed, you’ll need to set up a dedicated Python environment with the appropriate dependencies:

Create the environment: conda create -n mmyolo python=3.8 pytorch==1.10.1 torchvision==0.11.2 cudatoolkit=11.3 -c pytorch -y
Activate it with conda activate mmyolo
Install the required dependencies with a series of commands:

These commands install the MMYOLO framework and its dependencies, including PyTorch with CUDA support for GPU acceleration. The specific versions are important for compatibility with DJI’s requirements.

Dataset Configuration

Proper dataset configuration is crucial for successful model training. After copying your dataset to a folder within the MMYOLO repository (e.g., /mmyolo/data/your-project-name), you’ll need to modify the configuration file:

Open the yolov8_s_syncbn_fast_8xb16-500e_coco.py configuration file
Update the dataset paths to point to your specific data
Update train_ann_file to point to your annotation file
Point val_data_prefix to ‘dataset/dataset/valid/images’
Set the number of classes to 3 (must be less than 10 for DJI compatibility, but you can use more than we have if your dataset has more classes)

Define class names and a color palette for visualization:

class_name = ['pothole', 'crack', 'manhole']
metainfo = dict(
    classes=class_name,
    palette=[(20, 220, 60), (0, 0, 255), (255, 0, 0)]
)

Add metainfo=metainfo to both the training and validation dataloader dictionaries

This configuration tells the training process what classes to recognize and how to visualize them. The palette defines the colors used for bounding boxes during visualization, which helps differentiate between detected object classes.

Model Training

With the environment and dataset properly configured, you can start the training process with a single command:

This command launches the training process using a single GPU (indicated by the CUDA_VISIBLE_DEVICES=0 and the final “1” parameter). The training duration depends on your dataset size and GPU capability, but typically takes several hours.

During training, MMYOLO periodically saves checkpoints and logs performance metrics. The final trained model will be saved as a .pth file, which is the file you’ll submit to DJI for quantization.

Model Submission and Quantization

After successfully training the model, the next step is to upload it to DJI’s developer platform for quantization and optimization. This process involves:

Visiting the DJI developer website and navigating to the “Upload Model” section
Uploading the trained model file and providing calibration images for each object class
Setting our classes by Display Name

The DJI platform processes the model, optimizing it for deployment on the Matrice 4E. This quantization process reduces the model size and ensures it runs efficiently on the drone’s hardware.

Model Validation (Optional)

Although optional, validating the model on the DJI platform before deployment is recommended. This step allows you to verify that the model performs as expected with test images. The validation process involves:

Uploading a set of validation images
Setting a confidence threshold (typically around 50% but we set ours to 25%)
Reviewing the detection results

This validation step helps identify potential issues before deploying the model to the drone, saving time and frustration in the field. If everything works out, our validation images should turn up with some detected potholes.

Model Deployment to the Drone

Once the model has been quantized and validated, it can be downloaded from the DJI platform and deployed to the Matrice 4E. The deployment process is straightforward:

Download the template and assign the Aircraft SN
Download the optimized model file from the DJI platform
Connect the remote controller to a computer
Side load the model file to the remote controller
Use the DJI Pilot 2 app to install the model on the drone
1. Upload the File from the SD Card

After installation, the model can be activated through the DJI Pilot 2 interface, allowing for real-time object detection during flight. Users can select which object classes to detect and adjust confidence thresholds as needed.

When the model is fully loaded, click on Enter Camera View and you should receive the Preflight Check

On the right-hand side, you should now be able to check into the AI models and see our classes.

Real-World Performance and Considerations

When testing the deployed model in the field, our tests indicated flight height plays a crucial role in detection accuracy. Since their model was trained on images taken close to the ground, flying at higher altitudes reduced detection accuracy. This highlights the importance of training models on data collected from similar heights and angles as the intended deployment scenarios.

The detection results can be viewed in real-time through the DJI Pilot 2 app, allowing operators to immediately assess the model’s effectiveness. When an object is detected, a bounding box appears around it, with the object class and confidence level displayed. This immediate feedback is valuable for adjusting flight patterns or detection parameters on the fly.

For applications like infrastructure inspection, this real-time detection capability streamlines the workflow significantly. Rather than analyzing footage after the flight, inspectors can identify issues as they occur and focus on areas of interest immediately.

Conclusion

You've now learned how to transform your drone into an intelligent detection platform without building custom applications from scratch. Through this tutorial, you've discovered how to train custom YOLOv8 models using MMYOLO, navigate DJI's developer platform for model optimization, and deploy AI capabilities directly to the Matrice 4E using the standard Pilot 2 interface. This workflow is so easy to use, you can probably get started right away training your own custom model. Feel free to share with us your work and results on the meetup page.