Data Flow data processing sink as REST

Note

• This article is to show the functionality

Prerequisite

• Azure Account
• Azure synapse analytics workspace
• Azure logic app
• Azure Storage
• Upload the movidedb1.csv from this report in data folder to folder called movidedincoming
• create a new folder called moviesoutput

Create Logic app flow

• Create a new logic app
• Create a workflow

• Create a http trigger
• Select when Http request is received
• First delete the blob
• Save the body of request as file

containername: output
filename: output.json

Data Flow Activity

• Log into Azure synapse analytics workspace or studio
• First upload the data

Data Flow data processing lineage in Azure Purview using delta CRUD operation

Note

Prerequisite

• Azure Account
• Azure synapse analytics workspace
• Azure Purview
• Link both of those
• Provide Purview contributor to synapse analytics managed identity
• Azure Storage
• Upload the movidedb1.csv from this report in data folder to folder called movidedincoming
• create a new folder called moviesoutput

Data Flow Activity

• Log into Azure synapse analytics workspace or studio
• First upload the data

• Go to Develop and create a data flow
• Create a new dataflow
• Here is the entire flow

Using Azure Databricks anonymization of Text with PII

Use Case

• Ability to Anonymize PII data in a dataset
• Used for Data engineering
• used for Machine learning

Pre Requisite

• Azure Account
• Azure Storage account
• Azure Databricks
• install libraries

presidio-analyzer 
presidio-anonymizer

Reference

• PII Entities supported — https://microsoft.github.io/presidio/supported_entities/
• Repo for Presidio — https://github.com/microsoft/presidio/tree/main/docs/samples/deployments/spark

Code in Spark

• Confirm the above presidio libraries are installed

• Now lets write the code
• Bring all the imports

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities.engine import OperatorConfig
from pyspark.sql.types import StringType
from pyspark.sql.functions import input_file_name, regexp_replace
from pyspark.sql.functions import col, pandas_udf
import pandas as pd
import os

• Load the sample Titanic data set

df = spark.sql("Select * from default.titanictbl")

• Now display the…

How to implement Responsible AI features using titanic data set

Intro

Pre Requisite

• Azure account
• Azure storage
• Azure machine learning services

Code Sample

• Using notebook code to perform responsible ai features
• Explainability
• Fairness
• and more

library installation

!pip install raiwidgets==0.9.2
!pip install fairlearn==0.7.0

• Check the versions

!pip show fairlearn
!pip show raiwidgets

• Now the actual code
• Load the data from dataset or from file

from raiwidgets import ExplanationDashboardfrom azureml.core import Dataset
from azureml.data.dataset_factory import DataType# create a TabularDataset from a delimited file behind a public web url and convert column "Survived" to boolean
web_path ='https://dprepdata.blob.core.windows.net/demo/Titanic.csv'
titanic_ds = Dataset.Tabular.from_delimited_files(path=web_path, set_column_types={'Survived': DataType.to_bool()})# preview the first 3 rows of titanic_ds
titanic_ds.take(3).to_pandas_dataframe()
from azureml.core import Workspace, Datasetimport…

Azure Synapse analytics workspace gets lineage for copy activity in Azure Purview

Copy activity lineage in Azure Purview

Note

• This article is to show the functionality

Prerequisite

• Azure Account
• Azure synapse analytics workspace
• Azure Purview
• Link both of those
• Provide Purview contributor to synapse analytics managed identity
• Azure Storage
• Upload the titanic.csv from this report in data folder to folder called titianic
• create a new folder called titanicoutput

Copy Activity

• Create a new integrate pipeline
• Drag and drop copy activity
• For source select csv with ADLS gen2

• here is the configuration for source

Use Custom score script to deploy REST API

Steps

• Create a automated ML run
• Now navigate to best model
• Go to outputs/logs and select output and download model.pkl
• Download the score file also
• Edit the score file

# ---------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# ---------------------------------------------------------
import json
import logging
import os
import pickle
import numpy as np
import pandas as pd
import joblibimport azureml.automl.core
from azureml.automl.core.shared import logging_utilities, log_server
from azureml.telemetry import INSTRUMENTATION_KEYfrom inference_schema.schema_decorators import input_schema, output_schema
from inference_schema.parameter_types.numpy_parameter_type import NumpyParameterType
from inference_schema.parameter_types.pandas_parameter_type import PandasParameterType
from inference_schema.parameter_types.standard_py_parameter_type import StandardPythonParameterTypedata_sample = PandasParameterType(pd.DataFrame({"PassengerId": pd.Series([0], dtype="int64"), "Pclass": pd.Series([0], dtype="int64"), "Name": pd.Series(["example_value"], dtype="object"), "Sex": pd.Series(["example_value"], dtype="object")…

Enable Application insights logs for inferencing using REST API

Steps

• Every Azure ML workspace will have it’s own application insights
• Log into Azure portal
• Navigate to Application insights folder

Performance

Batch process only

Steps

• Create a Azure data factory
• Get table name and credentials for Azure SQL
• Azure SQL will be the data source input
• Output will also be saved in Azure SQL
• Create a new pipeline
• Get lookup activity
• in Azure SQL create a table and load sample data as below

Create table dbo.precribdata
(
  id int,
  ptext varchar(4000),
  inserttime datetime
)insert into dbo.precribdata (id, ptext, inserttime) values (1, 'Subject is taking 100mg of ibuprofen twice daily.', getdate())
insert into dbo.precribdata (id, ptext, inserttime) values (2, 'Subject is taking 100mg of ibuprofen twice daily.', getdate())
insert into dbo.precribdata (id, ptext, inserttime) values (3, 'Subject…