Calling Databricks notebook execution in ADF

We now have laid down everything to trigger the notebook execution in ADF. Going back to the factory, we're going to add a linked service. So far, all the linked services we created in this book were connected to a data store: SQL Server, blob storage, and so on. This time, we're going to use a computation linked service: Azure Databricks.

As shown in the following screenshot, add a linked service. Click on the Compute tab, select Azure Databricks, and click on Continue:

We'll now enter the details of the cluster in the next step. We used Azure Databricks for the Type property. The following screenshot shows the properties to set:

The properties are explained as follows:

• Connect via integration runtime: We use the Default one. It has access to all Azure resources.
• Account selection method: From Azure subscription.
• Select cluster: We're going to create a cluster on the fly and it will be running only for the duration of our job; therefore, we select...