Using Dataflow and Synapse Spark to analyze data and Spark ML modelling. Using TPCH data Pre-requisites Azure subscription Azure Storage Account Azure Synapse Analytics Load TPCH data I had to limit line items data as it was 150 billion rows. Dataset Rows Orders: 15,000,000,000 Customers: 1,500,000,000 Lineitems: 279,286,998 Goal Use Dataflow to analyze data Use join and create year, month and day columns

Process large data frame by chunks of 20 Pre-requisites Azure Account Storage account Azure machine learning Azure open ai service Goal Azure Open AI is a service that allows you to use GPT-3 to generate text. But there is limitation on how much we can send at a time. At the time of this document, it was 20 requests/sec. So…

Use Form Recognizer to extract Large text and chunk and summarize Pre-requisites Azure subscription Azure Machine Learning Workspace Document in pdf format Use python Code install azure ai form recognizer pip install azure.ai.formrecognizer import libraries from azure.core.credentials import AzureKeyCredential from azure.ai.formrecognizer import DocumentAnalysisClient import openai import re import requests import sys from num2words import num2words import os import pandas as pd import numpy as np from openai.embeddings_utils import get_embedding, cosine_similarity from transformers…

Azure Open AI Summarize in pandas data frame Overview Summarize the data in pandas data frame. Load PDF data into pandas data frame Clean the data. Load all pages into one row per pdf. Using Azure Machine learning Load the pdf in a blob container. Code Pip install pdfreader Load pip reader pip install pdfreader from pdfreader import SimplePDFViewer Import…

Efficient Embedding Search for Natural Language Queries Introduction use open ai and create embedding Search columns based on embeddings Demo application Restrict 40 rows to azure open ai API install packages pip install tiktoken pip install openai Code Create encoding for gpt2 to get tokens encoding = tiktoken.get_encoding("gpt2") Read the data import pandas as pd df = pd.read_json('sample.json')

Using AutoML to predict the failure of a machine Introduction Idea here is to predict the failure of a machine using AutoML AutoML can increase data science process productivity by automating time-consuming, iterative tasks Once the model is developed, we will deploy the model as a web service For deployment we are using Managed Online endpoint, Fully managed service that…

Execute Ray parallel processing in Azure Machine Learning with single node CPU Prerequisites Azure subscription Azure Machine Learning Workspace Azure Storage Account Install Ray library Code install ray libraries pip install -qU "ray[tune]" lightgbm_ray install pyarrow specific version for the code to work pip install pyarrow==6.0.1 Restart the kernel import libraries from typing import Tuple import ray from ray.train.batch_predictor import BatchPredictor from ray.train.lightgbm import LightGBMPredictor from ray.data.preprocessors.chain…

Let’s build a Power App to use Azure Open AI for various use cases. What’s needed. Register for Azure Open AI — https://learn.microsoft.com/en-us/azure/cognitive-services/openai/overview Once got approved create a azure open ai resource in Azure portal Select region as East US At the time of writing this article davinci-003 is only available in East US Create a deployment inside the resource Create a Power App

Pre-requisites Azure Account Azure Data Explorer Cluster Azure Service principal Create a secret Assign API permission for graph to user.read in delegate and application permissions Scope is only to move the meta data from microsoft graph to ADX Azure Machine Learning Workspace Create a compute cluster Code install libraries pip install azure-kusto-ingest …