Adding a Manual Migration Task

1. Introduction

Before reading this page, make sure you are familiar with the Async Task Architecture and the Process Api guides.

Manual Migration Tasks provide a framework for executing long-running data migration jobs in the background. They are designed for upgrade scenarios where existing data needs to be converted to a new format — for example, migrating file attachments from legacy storage to a new media object system.

Each Manual Migration Task:

  • Processes records in configurable batches to avoid overloading the server.

  • Tracks progress through three phases: Queueing, Processing, and Finalizing.

  • Supports item-level retry for transient failures.

  • Integrates with the Administration Panel so site administrators can trigger, monitor, and dismiss tasks.

Manual Migration Tasks are one concrete use of the Async Task system. They use the runner type manual-migration-tasks and their task records are stored in the manual_migration_tasks table.

2. Components of a Manual Migration Task

A Manual Migration Task requires three components:

Component Description

Task Handler

Implements AsyncTaskHandlerInterface (typically extends AbstractAsyncTaskHandler). Contains the migration logic — which records to fetch and how to process each one.

Process Handler (Action)

Implements ProcessHandlerInterface. Bridges the frontend "Run Migration" button to the async task dispatcher. Configures the process as async with the correct handler key and runner type.

Doctrine Migration

Seeds the task record into the manual_migration_tasks database table during upgrade. This is what makes the task appear in the admin UI.

3. Step-by-Step Guide

This section walks through creating a complete Manual Migration Task using a concrete example: migrating legacy Notes file attachments to the media object storage system.

The example below places code in core/backend/ because Manual Migration Tasks are typically part of the core platform (shipped with upgrades). If you are building an extension, place the handler and action under extensions/<your-extension>/backend/ and the migration under extensions/<your-extension>/backend/Migrations/ instead.

In the following subsection we are going to be looking into the implementation of a Manual Migration Task that migrates Notes attachments from legacy storage (files stored in upload/{note_id}) to the new media object system. This example is based on a real migration task implemented for SuiteCRM 8.10.0.

3.1 Step 1 — Create the Task Handler

The Task Handler contains the core migration logic. Extend AbstractAsyncTaskHandler for sensible defaults.

The core task handler: core/backend/ManualMigrations/Migrations/MigrateNotesFiles/MigrateNotesFilesTaskHandler.php:

<?php

namespace App\ManualMigrations\Migrations\MigrateNotesFiles;

use App\AsyncTask\Service\LegacyBridge\AsyncTaskLegacyHandler;
use App\AsyncTask\Service\TaskHandler\AbstractAsyncTaskHandler;
use App\AsyncTask\Service\TaskHandler\AsyncTaskBatchItem;
use App\Data\Entity\Record;
use App\Data\LegacyHandler\PreparedStatementHandler;
use App\Engine\Model\Feedback;
use App\MediaObjects\Repository\MediaObjectManagerInterface;
use App\MediaObjects\Services\LocalFileMediaObjectMigratorInterface;
use Psr\Log\LoggerInterface;

class MigrateNotesFilesTaskHandler extends AbstractAsyncTaskHandler
{
    public function __construct(
        protected AsyncTaskLegacyHandler $legacyHandler,
        protected PreparedStatementHandler $preparedStatementHandler,
        protected MediaObjectManagerInterface $mediaObjectManager,
        protected LocalFileMediaObjectMigratorInterface $migrator,
        protected LoggerInterface $logger
    ) {
    }

    public function getHandlerKey(): string
    {
        return 'migrate-notes-files';
    }

    public function getType(): string
    {
        return 'manual-migration-tasks';
    }

    public function getNextBatchToQueue(
        Record $task,
        array $progress,
        int $batchSize
    ): array {
        $offset = $progress['enqueue_offset'] ?? 0;

        try {
            $qb = $this->preparedStatementHandler->createQueryBuilder();
            $qb->select('id', 'filename', 'file_mime_type')
                ->from('notes')
                ->where('filename IS NOT NULL')
                ->andWhere("filename != ''")
                ->andWhere('deleted = 0')
                ->setFirstResult($offset)
                ->setMaxResults($batchSize);

            $results = $qb->executeQuery()->fetchAllAssociative();
        } catch (\Throwable $e) {
            $this->logger->error(
                'MigrateNotesFilesTaskHandler::getNextBatchToQueue failed: '
                . $e->getMessage(),
                ['component' => 'migrate-notes-files']
            );

            return [];
        }

        $items = [];
        foreach ($results as $row) {
            $items[] = new AsyncTaskBatchItem(
                $row['id'],
                [
                    'record_id' => $row['id'],
                    'filename' => $row['filename'],
                    'file_mime_type' => $row['file_mime_type']
                        ?? 'application/octet-stream',
                ]
            );
        }

        return $items;
    }

    public function processItem(Record $task, array $item): Feedback
    {
        $feedback = new Feedback();
        $data = $item['data'] ?? [];
        $recordId = $data['record_id'] ?? '';
        $filename = $data['filename'] ?? '';
        $mimeType = $data['file_mime_type'] ?? 'application/octet-stream';

        if (empty($recordId) || empty($filename)) {
            return $feedback->setSuccess(false)
                ->setMessages(['Missing record_id or filename']);
        }

        $this->legacyHandler->startLegacy();

        try {
            $legacyDir = $this->legacyHandler->getLegacyDir();
            $legacyPath = $legacyDir . '/upload/' . $recordId;

            // Check if already migrated
            $existing = $this->mediaObjectManager->getLinkedMediaObjects(
                'private-documents',
                'Notes',
                $recordId,
                'file'
            );

            if (!empty($existing)) {
                // Clean up legacy file if it still exists
                if (file_exists($legacyPath)) {
                    unlink($legacyPath);
                }

                return $feedback->setSuccess(true)
                    ->setMessages(['Already migrated']);
            }

            if (!file_exists($legacyPath)) {
                return $feedback->setSuccess(false)
                    ->setMessages(['Legacy file not found: upload/' . $recordId]);
            }

            // Perform the migration
            $record = $this->migrator->migrate(
                $legacyPath,
                'private-documents',
                $mimeType,
                $filename,
                $filename,
                'Notes',
                $recordId,
                'file'
            );

            $feedback->setSuccess(true);
            $feedback->setData(['media_object_id' => $record?->getId()]);
        } catch (\Throwable $e) {
            $this->logger->error(
                'MigrateNotesFilesTaskHandler::processItem failed: '
                . $e->getMessage(),
                [
                    'component' => 'migrate-notes-files',
                    'record_id' => $recordId,
                ]
            );
            $feedback->setSuccess(false)->setMessages([$e->getMessage()]);
        } finally {
            $this->legacyHandler->stopLegacy();
        }

        return $feedback;
    }
}

3.1.1 Handler Key and Type

getHandlerKey() returns a unique string (e.g. 'migrate-notes-files'). This key is stored in the task record’s service_key field and is used to match the task to its handler at runtime.

getType() must return 'manual-migration-tasks'. This tells the system to use the AsyncManualMigrationRunner.

3.1.2 getNextBatchToQueue()

This method is called repeatedly during the Queueing phase (see Queueing phase). It must:

  • Use $progress['enqueue_offset'] for pagination (the runner increments this automatically).

  • Return an array of AsyncTaskBatchItem objects, each with a unique key and a data payload.

  • Return an empty array when there are no more items to enqueue.

3.1.3 processItem()

Processes a single item. See the item array reference for the full $item structure.

Return a Feedback object:

  • setSuccess(true) for success, setSuccess(false) for failure.

  • setMessages([…​]) to attach status messages (first message stored as error_message on failure).

  • setData([…​]) to store result data on the item.

3.1.4 Optional Methods

The following AbstractAsyncTaskHandler defaults can be overridden:

Method Default Description

hasFinalization()

false

Return true if the handler needs a finalization phase (e.g. merging output files).

finalize(Record $task)

No-op (returns success)

Post-processing logic after all items are processed.

getMaxItemRetries()

1

Max automatic retries per failed item.

allowsFailureRetry()

false

Whether "Retry Failed" is available to administrators.

allowsFailureRerun()

false

Whether "Re-run" is available to administrators.

3.2 Step 2 — Create the Process Handler (Action)

The Process Handler connects the frontend "Run Migration" button to the async task system. It implements ProcessHandlerInterface and configures the process as asynchronous.

Create the file core/backend/ManualMigrations/Migrations/MigrateNotesFiles/MigrateNotesFilesAction.php:

<?php

namespace App\ManualMigrations\Migrations\MigrateNotesFiles;

use ApiPlatform\Metadata\Exception\InvalidArgumentException;
use App\Process\Entity\Process;
use App\Process\Service\ProcessHandlerInterface;

class MigrateNotesFilesAction implements ProcessHandlerInterface
{
    protected const MSG_OPTIONS_NOT_FOUND = 'Process options are not defined';
    protected const PROCESS_TYPE = 'migration-task-migrate-notes-files';

    public function getProcessType(): string
    {
        return self::PROCESS_TYPE;
    }

    public function requiredAuthRole(): string
    {
        return 'ROLE_USER';
    }

    public function getRequiredACLs(Process $process): array
    {
        $options = $process->getOptions();
        $module = $options['module'] ?? '';

        return [
            $module => [
                ['action' => 'view', 'record' => $options['id'] ?? ''],
            ],
        ];
    }

    public function validate(Process $process): void
    {
        if (empty($process->getOptions())) {
            throw new InvalidArgumentException(self::MSG_OPTIONS_NOT_FOUND);
        }

        $options = $process->getOptions();
        if (empty($options['module']) || empty($options['id'])) {
            throw new InvalidArgumentException(self::MSG_OPTIONS_NOT_FOUND);
        }
    }

    public function configure(Process $process): void
    {
        $options = $process->getOptions();
        $process->setId($options['id'] ?? '');
        $process->setAsync(true);
        $process->setAsyncHandlerKey('migrate-notes-files');
        $process->setAsyncRunnerType('manual-migration-tasks');
        $process->setModule($options['module'] ?? '');
    }

    public function run(Process $process): void
    {
        // Empty — async processes are handled by the runner
    }
}

3.2.1 Process Type Convention

The process type is constructed by the frontend from the viewdef action configuration. The detail view metadata defines a run-migration action with asyncProcessKeyPrefix set to 'migration-task' and asyncProcessKeyField set to 'service_key'. At runtime:

{asyncProcessKeyPrefix}-{service_key_value}
→ migration-task-migrate-notes-files

The PROCESS_TYPE constant in your action class must match this constructed value exactly.

3.2.2 configure() Method

The configure() method sets the async properties that tell the ProcessProcessor how to dispatch the task (see Process API integration):

  • setAsync(true) — marks the process as asynchronous.

  • setAsyncHandlerKey('migrate-notes-files') — must match the Task Handler’s getHandlerKey().

  • setAsyncRunnerType('manual-migration-tasks') — must match the Task Handler’s getType().

3.2.3 run() Method

Left empty for async processes. The ProcessProcessor detects async=true and dispatches the task to the Messenger bus instead of calling run().

3.3 Step 3 — Create the Doctrine Migration

The Doctrine migration seeds the task record into the manual_migration_tasks table. This migration runs automatically during the upgrade process.

If you are building an extension, place the migration under extensions/<your-extension>/backend/Migrations/ with the namespace App\Extension\<dirName>\Migrations. Extension migrations are auto-discovered via ExtensionMigrationsExtension — no additional configuration is needed.

Create the file core/backend/Migrations/Version20260217120000.php (use a timestamp-based name following the existing convention):

<?php

declare(strict_types=1);

namespace App\Migrations;

use Doctrine\DBAL\Schema\Schema;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;

final class Version20260217120000 extends BaseMigration
    implements ContainerAwareInterface
{
    public function getDescription(): string
    {
        return 'Add Migrate Notes Attachments manual migration task';
    }

    public function up(Schema $schema): void
    {
        $this->log(
            'Migration Version20260217120000: '
            . 'Adding Migrate Notes Attachments task'
        );

        $serviceKey = 'migrate-notes-files';

        // Guard: skip if already exists
        $existing = $this->connection->fetchOne(
            'SELECT COUNT(*) FROM manual_migration_tasks '
            . 'WHERE service_key = ? AND deleted = 0',
            [$serviceKey]
        );

        if ((int)$existing > 0) {
            $this->log(
                'Migration Version20260217120000: '
                . 'Task already exists, skipping'
            );
            return;
        }

        $now = date('Y-m-d H:i:s');

        $this->connection->insert('manual_migration_tasks', [
            'id' => create_guid(),
            'name' => 'Migrate Notes Attachments',
            'type' => 'background',
            'status' => 'initial',
            'service_key' => $serviceKey,
            'description' => 'Migrates Notes file attachments from '
                . 'legacy storage (upload/{note_id}) to the new '
                . 'media object storage system.',
            'date_entered' => $now,
            'date_modified' => $now,
            'created_by' => '1',
            'modified_user_id' => '1',
            'assigned_user_id' => '1',
            'deleted' => 0,
            'allow_failure_retry_action' => 1,
            'allow_failure_rerun_action' => 0,
        ]);

        $this->log(
            'Migration Version20260217120000: Task added successfully'
        );
    }

    public function down(Schema $schema): void
    {
        $this->connection->executeStatement(
            'DELETE FROM manual_migration_tasks WHERE service_key = ?',
            ['migrate-notes-files']
        );
    }
}

3.3.1 Important Fields

Field Description

id

A UUID. Use create_guid() to generate one.

name

Human-readable name displayed in the UI.

type

Set to 'background' for tasks that run via the Messenger worker.

status

Must be 'initial' for new tasks. This is the only status that shows the "Run Migration" button.

service_key

Must match the value returned by your Task Handler’s getHandlerKey().

description

A description shown in the task detail view.

allow_failure_retry_action

Set to 1 if your handler’s allowsFailureRetry() returns true.

allow_failure_rerun_action

Set to 1 if your handler’s allowsFailureRerun() returns true.

Always include a guard clause that checks whether the task record already exists before inserting. Re-running Doctrine migrations must not create duplicate records.

3.4 Step 4 — Clear the Cache

After creating all the files, clear the Symfony cache so the new services are discovered:

php bin/console cache:clear

No additional service registration is needed. Both AsyncTaskHandlerInterface and ProcessHandlerInterface are auto-configured via tagged interfaces in config/core_services.yaml.

4. Conventions and Gotchas

4.1 Handler Key and Process Type Must Be Consistent

Three values must be consistent across your components:

  1. The Task Handler’s getHandlerKey() return value (e.g. 'migrate-notes-files').

  2. The Doctrine Migration’s service_key column value — must match the handler key.

  3. The Process Handler’s PROCESS_TYPE — must follow the pattern {asyncProcessKeyPrefix}-{service_key} (e.g. 'migration-task-migrate-notes-files').

If any of these are mismatched, the system will not be able to resolve the correct handler and the task will fail silently.

4.2 The run() Method Must Be Empty for Async Handlers

The ProcessProcessor detects async=true from the configure() method and dispatches the task to the Messenger bus. It does not call run(). Leave the run() method empty.

4.3 Always Guard Against Duplicate Records in Migrations

Doctrine migrations must be idempotent. Always check whether the task record already exists before inserting. The recommended pattern is to query by service_key and deleted = 0.

4.4 Worker Must Run as the Web Server User

When the task handler writes files (uploads, media objects), the Messenger worker must run as the same user as the web server (e.g. www-data). Otherwise, file permission errors will occur.

4.5 Use the Legacy Bridge Carefully

If your handler accesses legacy SuiteCRM code (via AsyncTaskLegacyHandler or any LegacyHandler subclass), always call startLegacy() before and stopLegacy() after (in a finally block). The legacy bridge depends on working directory and session state management.

4.6 Extension Placement

For tasks shipped as part of an extension, place files under the extension directory:

  • Task Handler: extensions/<ext>/backend/ManualMigrations/Migrations/<Name>/<Name>TaskHandler.php

  • Process Handler: extensions/<ext>/backend/ManualMigrations/Migrations/<Name>/<Name>Action.php

  • Doctrine Migration: extensions/<ext>/backend/Migrations/Version<timestamp>.php

Extension migrations are auto-discovered via ExtensionMigrationsExtension and use the namespace App\Extension\<dirName>\Migrations.

5. Summary Checklist

When creating a new Manual Migration Task, ensure you have:

  1. [ ] Created a Task Handler class extending AbstractAsyncTaskHandler with:

    • getHandlerKey() returning a unique key.

    • getType() returning 'manual-migration-tasks'.

    • getNextBatchToQueue() returning paginated AsyncTaskBatchItem arrays.

    • processItem() returning a Feedback object for each item.

  2. [ ] Created a Process Handler (Action) class implementing ProcessHandlerInterface with:

    • getProcessType() matching the pattern migration-task-{handler-key}.

    • configure() setting async=true, asyncHandlerKey, and asyncRunnerType='manual-migration-tasks'.

    • An empty run() method.

  3. [ ] Created a Doctrine Migration that inserts the task record into manual_migration_tasks with:

    • service_key matching your handler key.

    • status set to 'initial'.

    • An idempotency guard (check before insert).

    • Correct allow_failure_retry_action / allow_failure_rerun_action flags matching your handler.

  4. [ ] Cleared the cache with php bin/console cache:clear.

  5. [ ] Tested with the Messenger worker running as the web server user, see the Messenger Setup guide.

Content is available under GNU Free Documentation License 1.3 or later unless otherwise noted.