How to run batch training at scale and parallize

Use case

code

from azureml.core import Workspace
ws = Workspace.from_config()
from azureml.core import Workspace, Datasetsubscription_id = 'xxxxxxxxxxxxxxxxxxxxx'
resource_group = 'xxxxxxxxxxxxxxx'
workspace_name = 'xxxxxxxxxxxxxxxx'
workspace = Workspace(subscription_id, resource_group, workspace_name)dataset = Dataset.get_by_name(workspace, name='TitanicTraining')
data = dataset.to_pandas_dataframe()
from azureml.core.datastore import Datastoreaccount_name = "storageacctname"
datastore_name="titanicstore"
container_name="titanic"

Data Flow data processing sink as REST

Note

Prerequisite

Create Logic app flow

containername: output
filename: output.json

Data Flow Activity


Data Flow data processing lineage in Azure Purview using delta CRUD operation

Note

Prerequisite

Data Flow Activity


Using Azure Databricks anonymization of Text with PII

Use Case

Pre Requisite

presidio-analyzer 
presidio-anonymizer

Reference

Code in Spark

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities.engine import OperatorConfig
from pyspark.sql.types import StringType
from pyspark.sql.functions import input_file_name, regexp_replace
from pyspark.sql.functions import col, pandas_udf
import pandas as pd
import os
df = spark.sql("Select * from default.titanictbl")


How to implement Responsible AI features using titanic data set

Intro

Pre Requisite

Code Sample

library installation

!pip install raiwidgets==0.9.2
!pip install fairlearn==0.7.0
!pip show fairlearn
!pip show raiwidgets
from raiwidgets import ExplanationDashboardfrom azureml.core import Dataset
from azureml.data.dataset_factory import DataType
# create a TabularDataset from a delimited file behind a public web url and convert column "Survived" to boolean
web_path ='https://dprepdata.blob.core.windows.net/demo/Titanic.csv'
titanic_ds = Dataset.Tabular.from_delimited_files(path=web_path, set_column_types={'Survived': DataType.to_bool()})
# preview the first 3 rows of titanic_ds
titanic_ds.take(3).to_pandas_dataframe()
from azureml.core import Workspace, Dataset
import…


Azure Synapse analytics workspace gets lineage for copy activity in Azure Purview

Copy activity lineage in Azure Purview

Note

Prerequisite

Copy Activity


Use Custom score script to deploy REST API

Steps

# ---------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# ---------------------------------------------------------
import json
import logging
import os
import pickle
import numpy as np
import pandas as pd
import joblib
import azureml.automl.core
from azureml.automl.core.shared import logging_utilities, log_server
from azureml.telemetry import INSTRUMENTATION_KEY
from inference_schema.schema_decorators import input_schema, output_schema
from inference_schema.parameter_types.numpy_parameter_type import NumpyParameterType
from inference_schema.parameter_types.pandas_parameter_type import PandasParameterType
from inference_schema.parameter_types.standard_py_parameter_type import StandardPythonParameterType
data_sample = PandasParameterType(pd.DataFrame({"PassengerId": pd.Series([0], dtype="int64"), "Pclass": pd.Series([0], dtype="int64"), "Name": pd.Series(["example_value"], dtype="object"), "Sex": pd.Series(["example_value"], dtype="object")…


Enable Application insights logs for inferencing using REST API

Steps

Performance


Batch process only

Steps

Create table dbo.precribdata
(
id int,
ptext varchar(4000),
inserttime datetime
)
insert into dbo.precribdata (id, ptext, inserttime) values (1, 'Subject is taking 100mg of ibuprofen twice daily.', getdate())
insert into dbo.precribdata (id, ptext, inserttime) values (2, 'Subject is taking 100mg of ibuprofen twice daily.', getdate())
insert into dbo.precribdata (id, ptext, inserttime) values (3, 'Subject…


Azure Data/AI platform architecture — Analytics

Build your data strategy based on Data governance, Data Life cycle management and ML/AI Data science life cycle management, all in PaaS with Security in mind.

Use Case

Architecture

Explained

Data Source

Ingestion

Balamurugan Balakreshnan

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store