Using Azure Cognitive Services Speech to Text and Logic apps

No Code — Workflow style

please use this as reference, as this might not be your exact use case. But you can see how AI can help drive insights and automate large audio dataset

Pre requisite

• Azure Account
• Azure Storage account
• Azure Cognitive Services
• Azure Logic apps
• Get connection string for storage
• Get the primary key to be used as subcription key for cognitive services
• Audio file should be wav format
• Audio file cannot be too big
• Audio time 10 min

Logic apps

• First create a trigger from Blob

• Create a connection string using blob connection string

Using Azure Cognitive Services Forms Recognizer and Logic apps

No Code — Workflow style

Pre requisite

• Azure Account
• Azure Storage account
• Azure Cognitive Services — Form Recognizer
• Azure Logic apps
• Get connection string for storage
• Get the primary key to be used as subcription key for cognitive services
• Audio file should be pdf format
• Audio file cannot be too big

Full Flow

Logic apps

• First create a trigger from Blob

Using Azure Cognitive Services Speech to Text and Logic apps

No Code — Workflow style

We can re use the same pattern for other Azure Cognitive services as well.

Pre requisite

• Azure Account
• Azure Storage account
• Azure Cognitive Services
• Azure Logic apps
• Get connection string for storage
• Get the primary key to be used as subscription key for cognitive services
• Audio file should be wav format
• Audio file cannot be too big
• Audio time 10 min

Logic apps

• First create a trigger from Blob

• Create a connection string using blob connection string

Using open census library to push error logs to Azure monitor

Prerequisite

• Azure account
• Azure Synapse workspace
• Azure Storage

Steps

• Create a Spark cluster
• install library
• create a conda file and upload
• Create a environment.yml

name: example-environment
channels:
  - conda-forge
dependencies:
  - python
  - numpy
  - pip
  - pip:
    - opencensus-ext-azure

• upload the yml file
• Create a notebook

Code

• Choose python as language

import logging
from opencensus.ext.azure.log_exporter import AzureLogHandlerlogger = logging.getLogger(__name__)# TODO: replace the all-zero GUID with your instrumentation key.
logger.addHandler(AzureLogHandler(
    connection_string='InstrumentationKey=xxxxx-xxxxxx-xxxxxx-xxxxxxx')
)

• now log some sample logs

logger.warning("Sample from open census test 01")
logger.error("Sample from open census test 02")

• NOw lets log an exception

from opencensus.ext.azure.trace_exporter import AzureExporter
from opencensus.trace.samplers import ProbabilitySampler
from…

Using open census library to push error logs to Azure monitor

Prerequisite

• Azure account
• Azure Databricks
• Azure Storage

Steps

• Create a Databricks cluster
• install library

opencensus-ext-azure

Spark code to predict classification for practical dataset — titanic survivor dataset

Use Case

• Use Titanic dataset
• Dataset is available in this folder called Titanic.csv
• Upload the data set to Azure storage

Prerequisite

• Azure account
• Azure storage account
• Create a container called titanic and upload the Titanic.csv file to container
• Create Azure Data bricks resource
• Create a azure key vault
• Store the primary key into a secret
• Configure the keyvault to azure databricks

Code

• First lets include the libraries

from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import IndexToString, StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.sql import functions as f

• Load the secrets from keyvault

storagekey = dbutils.secrets.get(scope = "allsecrects", key = "storagekey")

• …

Build a end to end data science pipeline lab using Azure data factory and azure databricks

Pre requisites

• Azure account
• Azure data factory
• Azure databricks
• Azure Storage ADLS gen2 — to store all the parquet file — data lake
• Azure Keyvault for storing secrets

End to End Pipeline Architecture

Steps

• For the notebooks i am using existing notebooks from microsoft doc web site
• the business logic here has not real business value
• each tasks are not related to each other, it’s to show the flow
• data used are public data sets

Components

• Data flow to show case Data warehouse — Facts/Dimension model
• Notebook — PassingParameters — to show case how to pass pipeline parameters into notebooks
• Notebook — DataEngineering — to show case data engineering…

When code checked trigger CI/CD using Azure DevOps

Prerequisite

• Azure Account
• Azure Machine learning
• Create a compute instance
• Create a compute cluster as cpu-cluster
• Select Standard D series version
• Create Train file to train the model
• Create a pipeline file to run the as pipeline

Steps

Create Train file as train.py

• Create a directory ./train_src
• Create a train.py
• Should be a python file not notebook

# Copyright (c) Microsoft. All rights reserved.
# Licensed under the MIT license.

import argparse
import os
import pandas as pd
import numpy as np
from azureml.core import Workspace, Dataset
from azureml.core import Dataset
from azureml.data.dataset_factory import DataType
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

import sklearn as sk
import pandas as pd
# import seaborn as sn
# import matplotlib.pyplot as plt
from sklearn.preprocessing import…