Azure Data Factory

About Azure Data Factory

Azure Data Factory is Microsoft's managed Azure service for hybrid ETL, ELT, data movement and workflow orchestration across cloud and on-premises systems. It is designed for teams that need repeatable pipelines, broad connector coverage, private-network access and managed execution without building a custom integration platform. Azure Data Factory remains an active Azure service. Microsoft now positions Data Factory in Microsoft Fabric as the next generation and provides assessment, mounting and migration tooling for organizations that want to move gradually rather than replace production pipelines in one step.

What is Azure Data Factory used for?

Azure Data Factory is used to build pipelines that collect, move, transform and publish data across many systems. A pipeline groups activities into a workflow, datasets describe the data being used, and linked services define connections to data stores or compute services. Common uses include scheduled ingestion, file-triggered workflows, migration projects, data-lake loading, warehouse preparation and repeatable movement between cloud and on-premises environments.

Data Factory can copy data directly between supported sources and destinations or dispatch transformation work to services such as Azure Databricks and Azure SQL Database. Mapping data flows provide a visual transformation model that runs on managed Spark compute, while control-flow activities coordinate sequencing, conditions, loops, lookups and external execution.

How do Azure Data Factory integration runtimes work?

The integration runtime is the execution bridge between pipeline activities and linked services. Microsoft documents three integration runtime types for Azure Data Factory: Azure integration runtime, self-hosted integration runtime and Azure-SSIS integration runtime.

Azure integration runtime provides managed compute for cloud data movement, mapping data flows and activity dispatch. Self-hosted integration runtime is installed in a private network and is used when cloud pipelines must reach on-premises or privately exposed data stores. Azure-SSIS integration runtime runs SQL Server Integration Services packages on managed Azure compute. Runtime choice affects network design, maintenance responsibility, performance and cost, so it should be selected around source location and security boundaries rather than after pipelines are already built.

What current Azure Data Factory limits should buyers plan around?

Microsoft currently documents a hard limit of 5,000 total entities in one data factory, including pipelines, datasets, triggers, linked services, private endpoints and integration runtimes. It also documents up to 10,000 concurrent pipeline runs per data factory, shared across all pipelines in that factory. Azure-SSIS integration runtimes have a default subscription quota of 64 CPU cores, with increases handled through the Azure quota process.

Activity-level limits also matter. The Lookup activity currently returns at most 5,000 rows, supports up to 4 MB of output and has a maximum duration of 24 hours. Large control datasets may therefore need pagination, staging or a two-level pipeline pattern instead of one oversized lookup.

What is changing with trusted-service firewall access?

Microsoft recommends the Modern trusted-service mode for Azure Data Factory and plans to retire Legacy trusted-service access in August 2027. The change matters for factories that use managed identities together with trusted Microsoft services firewall exceptions to reach Azure Storage or Azure Key Vault.

Affected patterns include self-hosted integration runtime access to Storage, Azure-SSIS integration runtime scenarios, REST linked services, Web activities and Azure Function activities that reach Storage or Key Vault through managed identity and a trusted-service firewall bypass. Microsoft recommends moving nonproduction factories first, validating connectivity, and then updating production. Depending on the design, teams may need IP allowlists, a Data Factory managed virtual network, a customer-owned virtual network, service endpoints or private endpoints. Buyers should treat this as a security and continuity migration because legacy-dependent connections may stop working after the August 2027 retirement.

How is Azure Data Factory priced?

Azure Data Factory does not have one flat product price. Microsoft meters different parts of the service separately, including orchestration and execution, integration runtime usage, data movement, mapping data-flow execution and debugging, and Data Factory read, write and monitoring operations. Azure-SSIS integration runtime is priced separately by the managed compute configuration used to host SSIS packages.

This means cost depends on pipeline frequency, activity count, runtime choice, data volume, transformation duration and any other Azure service invoked by the pipeline. Mapping data flows also incur managed compute and supporting storage charges while running and debugging. Pricing was rechecked on September 5, 2026. Buyers should use the Azure pricing calculator for region- and agreement-specific estimates rather than assuming one universal monthly price.

What are the main limitations and operational tradeoffs?

Data Factory reduces the need to build custom orchestration and data-movement services, but it does not remove the need for data engineering discipline. Complex pipelines can become difficult to troubleshoot when they contain many linked activities, parameters, triggers and external dependencies. Connector throttling, source-system limits and external compute failures can become the real bottleneck even when the Data Factory service itself is healthy.

Hybrid deployments add customer responsibility because self-hosted integration runtime runs on infrastructure the customer manages. Teams must plan patching, availability, outbound connectivity and capacity for those hosts. Mapping data flows also bring separate managed compute cost, so they are not automatically cheaper than transformations already running efficiently on another platform.

How does Azure Data Factory compare with Data Factory in Microsoft Fabric?

Microsoft describes Data Factory in Microsoft Fabric as the next generation of Azure Data Factory. Fabric combines data integration with a wider analytics environment and includes newer experiences such as Copilot-assisted pipeline and dataflow authoring. Existing Azure Data Factory workloads can continue running, but new Microsoft-centric analytics programs should compare Fabric Data Factory before expanding a large new ADF estate.

Microsoft also supports adding an existing Azure Data Factory as an item in a Fabric workspace. This does not migrate or modify the source factory. The ADF interface remains available and the pipelines, activities and integration runtimes still execute in Azure. Mounting is best treated as a visibility and transition step, not as a completed migration.

How does the 2026 Azure Data Factory to Fabric migration flow work?

Microsoft's current migration flow starts with a read-only assessment in Azure Data Factory. Pipelines and activities are classified as Ready, Needs review, Coming soon or Not compatible. The results can be exported to CSV so teams can plan remediation before changing production workloads.

After assessment, a factory can be mounted to a Fabric workspace and selected pipelines can be migrated. Linked services must be mapped to Fabric connections. Pipelines can still migrate when connections are not mapped, but dependent activities are deactivated until those connections are configured. Schedule triggers are migrated but left disabled by design, while other trigger types must be recreated and re-enabled after validation. Global parameters require separate treatment, and unsupported activities may need redesign rather than automated conversion.

Mapping Data Flow migration is currently in public preview. Microsoft converts supported mapping data flows into mapping data flow transforms in Dataflow Gen2. Teams should validate converted logic, connections, outputs and schedules in nonproduction before cutover. For scripted or bulk migrations, Microsoft also provides the Microsoft.FabricPipelineUpgrade PowerShell module, but the built-in migration experience is the recommended path for most migrations.

What architecture differences matter when moving from ADF to Fabric?

A Fabric migration is not a simple resource copy because several concepts differ. ADF datasets are reusable objects, while Fabric commonly places dataset properties directly inside activities. ADF global parameters map to a different Fabric variable-library pattern. ADF managed identity and Fabric workspace identity are not the same operating model, and Key Vault integration and connection behavior should be reviewed rather than assumed to be identical.

Scheduling also differs. ADF can reuse triggers across pipelines, while Fabric currently manages schedules at the pipeline level. Teams using dynamic linked-service patterns, metadata-driven frameworks, custom activities, SSIS, CDC or advanced trigger logic should review feature parity before committing to a cutover plan.

Who is Azure Data Factory best suited for?

Azure Data Factory is a strong fit for organizations that need managed batch data movement and orchestration across many systems, especially where hybrid connectivity, Azure networking or existing SSIS investments matter. It is also useful when teams want one Azure-native control plane for pipelines that call multiple storage, database and compute services.

It remains especially relevant for established environments with self-hosted integration runtimes, Azure-SSIS, private networking or extensive deployment automation that would make an immediate migration expensive. Those teams can keep operating ADF while assessing Fabric readiness and addressing migration gaps over time.

Who should choose something else?

A new analytics team building around Microsoft Fabric should evaluate Fabric Data Factory before creating a large new Azure Data Factory estate. Teams whose main requirement is code-first transformation or stream processing may be better served by the compute platform already running those workloads, such as Databricks or another orchestration system.

Simple application workflows may fit Azure Logic Apps better, while developers needing event-driven custom code should consider Azure Functions. Azure Data Factory is strongest when the core problem is repeatable data movement and orchestration across diverse data systems, especially when hybrid connectivity or an existing ADF and SSIS footprint is important.