Automated ML for vision/Deep learning

Prerequisite

• Azure Account
• Azure Storage
• Azure Machine learning Service
• Github account and repository
• Azure Service Principal Account
• Provide service principal contributor access to Machine learning resource
• Azure Keyvault to store secrets
• Update the keyvault with Service principal Secrets
• This automates the training code and Registers the model

Steps

• Log into ml.azure.com
• Start…

Protect PII by encrypting using Fernet in Azure Synapse Spark

Using fernet to encrypt key — symmetric encryption

Prerequisite

• Azure Account
• Azure Storage
• Azure Synapse Analytics workspace with Spark

Use Case

• Use encryption to encrypt PII or other sensitive data
• Data should be stored encrypted
• Only folks who have access to key can decrypt
• encrypt column level so only necessary columns can be encrypted and other’s are available for reporting
• here…

Using fernet to encrypt key — symmetric encryption

Prerequisite

• Azure Account
• Azure Storage
• Azure Databricks

Use Case

• Use encryption to encrypt PII or other sensitive data
• Data should be stored encrypted
• Only folks who have access to key can decrypt
• encrypt column level so only necessary columns can be encrypted and other’s are available for reporting
• here is the open source…

How to run batch training at scale and parallelize

Usecase

• Run models in parallel for training
• Scale based on data size

code

• First create a storage account
• Create a directory called titanic
• upload atleast 2 Tiantic.csv from data folder. For second file copy and rename as Titanic1.csv
• Now create a notebook

from azureml.core import Workspace
ws = Workspace.from_config()from azureml.core import…

Architecture

• Bonsai platform allows us to automate control system operations
• By designing the control system using engineering tool to simulate the control system.
• Based on the simulation bonzai training platform allows us to learn and build a model
• Details available — https://docs.microsoft.com/en-us/learn/autonomous-systems/intro-to-project-bonsai/1-introduction
• i believe this is the next generation of automation…

How to run batch training at scale and parallize

Use case

• Run models in parallel for training
• Scale based on data size

code

• First create a storage account
• Create a directory called titanic
• upload atleast 2 Tiantic.csv from data folder. For second file copy and rename as Titanic1.csv
• Now create a notebook

from azureml.core import Workspace
ws = Workspace.from_config()from azureml.core import…

Data Flow data processing sink as REST

Note

• This article is to show the functionality

Prerequisite

• Azure Account
• Azure synapse analytics workspace
• Azure logic app
• Azure Storage
• Upload the movidedb1.csv from this report in data folder to folder called movidedincoming
• create a new folder called moviesoutput

Create Logic app flow

• Create a new logic app
• Create a workflow

• Create a http trigger
• Select…

Data Flow data processing lineage in Azure Purview using delta CRUD operation

Note

Prerequisite

• Azure Account
• Azure synapse analytics workspace
• Azure Purview
• Link both of those
• Provide Purview contributor to synapse analytics managed identity
• Azure Storage
• Upload the movidedb1.csv from this report in data folder to folder called movidedincoming
• create a new folder called moviesoutput

Data Flow Activity

• Log into Azure synapse analytics workspace or studio
• First…