Azure Purview is now part of Microsoft Purview

I was ready for a nice relaxing evening today, when an email appeared in my inbox “Azure Purview is now Microsoft Purview!”

Initially I thought… yeah.. “just another Microsoft product name renaming” .. but when I read through it more in depth I found out, that this is NOT just a renaming.

The email I found in my inbox

Microsoft Purview = Azure Purview + Microsoft 365 Compliance portfolio

Microsoft Purview is the (new) name for a comprehensive set of products to govern, protect and manage your entire data estate. From tracking your data sources and their dependencies to managing data compliance regulations, data loss and data risks.

According to https://www.microsoft.com/en-us/security/business/microsoft-purview, Microsoft Purview helps you to

• Understand and govern data
• Safeguard data, wherever it lives and
• Improves risk and compliance posture

source: https://www.microsoft.com/en-us/security/business/microsoft-purview

You can also watch the intro-video of the Microsoft Purview team:

Can you tell me your name please?

Microsoft Purview is no new service or product, it combines already existing services under one single umbrella. For a better reference I reference the new-name-cheatsheet from https://www.microsoft.com/security/blog/2022/04/19/the-future-of-compliance-and-data-governance-is-here-introducing-microsoft-purview/

The new names of the pieces that put together Microsoft Purview.
Source: https://www.microsoft.com/security/blog/2022/04/19/the-future-of-compliance-and-data-governance-is-here-introducing-microsoft-purview/

Not TL;TR – if you need more information

• The announcement post by Jessica Hawk: https://azure.microsoft.com/en-us/blog/azure-purview-is-now-microsoft-purview/
• The future of compliance and data governance is here: Microsoft Purview (by Alym Rayani) – https://www.microsoft.com/security/blog/2022/04/19/the-future-of-compliance-and-data-governance-is-here-introducing-microsoft-purview/
• Learn more about Microsoft Purview: https://aka.ms/microsoftpurview
• If you already know or use Azure Purview -> the Microsoft Purview governance portal (aka Purview Studio) – https://purview.microsoft.com/governance – is your friend
• If you want to learn more about the compliance part of Microsoft Purview, the Microsoft Purview compliance portal – https://purview.microsoft.com/compliance – is your friend.

What’s next?

Well, this announcement needs some time on my side to get a better overview and deeper insight about the compliance part of what is now called Microsoft Purview. Stay tuned for some more blog posts and maybe some videos about it.

Love your data, #DoDataBetter and join the #TeamDataGovernance,

Wolfgang

Azure Purview Announcements March 2022

Hi members of #TeamDatagovernance!

It’s SQLBits week and I have the feeling that the Azure Purview team released some new features I want to summarize and give you some pointers to the announcement posts and documentation:

Dynamic SQL Lineage

For me, Data lineage is one of those fascinating techniques to better understand your data estate and get a better knowledge how systems are connected and what data flows are there in your data landscape.

Lineage was there in Azure Purview since the beginning (Azure Data Factory, SSIS lineage, Power BI) but this week another very important part of data lineage was put into public preview: Dynamic Lineage Extraction from Azure SQL Databases.

There are different ways how data lineage can be extracted from systems – one of them is static code analysis. The static approach includes all the CREATE PROCEDURE / CREATE VIEW statements and summarizes them into a lineage graph. This approach is powerful but there is more going on than the initially defined DDL statements like the execution of dynamic SQL statements.

The approach that Azure Purview implements is the dynamic lineage extraction (announcement post) that incorporates all the actions that happen in the database.

static versus runtime/dynmic analysis (source: https://azure.microsoft.com/en-us/blog/introducing-dynamic-lineage-extraction-from-azure-sql-databases-in-azure-purview/)

I’ve recorded a short video (youtube link) for a first, quick look on that feature.

Turn on lineage extraction in scan options for Azure SQL databases

Data lineage including a Stored proc

If you want to read more about the feature, learn what pre-requisites have to be met -> https://docs.microsoft.com/en-us/azure/purview/register-scan-azure-sql-database?tabs=sql-authentication#lineagepreview

Azure Purview now understand SAP Business Warehouse (BW)

There are new built-in sources added every now and then.. and today, SAP Business Warehouse (BW) was added to the list of supported sources:

https://techcommunity.microsoft.com/t5/azure-purview-blog/azure-purview-adds-support-for-sap-business-warehouse/ba-p/3253404

SAP BW and Purview documentation: https://docs.microsoft.com/en-us/azure/purview/register-scan-sap-bw

Workflow support for Data Access Request (Preview)

To know your data sources, their connections and so on is a beginning for your data journey, but access securement is a huge and very important topic.

The Purview team released a new block of functionality – Workflow support for managing data access in Azure Purview

I did not have any time to dive into that feature, but you will get an update (blog post, video) in the next weeks.

In the meantime, please have a look at the announcement blog posts from the team:

• Workflows in Azure Purview: https://docs.microsoft.com/en-us/azure/purview/concept-workflow

• https://techcommunity.microsoft.com/t5/azure-purview-blog/now-in-public-preview-azure-purview-workflows-for-managing-data/ba-p/3250703
• https://techcommunity.microsoft.com/t5/azure-purview-blog/now-in-public-preview-self-service-data-access-workflows-for/ba-p/3250513
• Approval workflows for business glossary: https://techcommunity.microsoft.com/t5/azure-purview-blog/now-in-public-preview-approval-workflow-for-business-glossary/ba-p/3250593

Happy data cataloguing and data scanning,

Wolfgang

SQLBits 2022 – It’s quite a #DataGovernance theme for me…

17 Days to go… Starting on Tuesday, 8th March London will be the Microsoft Data capital for a week. It’s SQLBits (https://arcade.sqlbits.com/) time again and this year – if everything goes well – it will be an in-person conference experience for me!

The conference itself is a hybrid event – so we can meet in-person or virtually!

I am really looking forward to a week full of data platform sessions, discussions and news, new features and content!

This years SQLBits will be different for me – It will be a full packed week – all in all I am there for

• 1 Traing Day about Data Governance – From Theory to Hands-On with Azure Purview on Tuesday
• 1 surprises on Friday
• 1 20 minutes Session (Friday): “Why you should care about Data Community and how the community cards about you”
• 1 50 minutes Session (Friday): “Data Governance with Azure Purview – Ask the Experts” Together with Victoria (t), Erwin (t) and Richard (t) we will answer and discuss your Data Governance / Azure Purview questions. Submit your questions here: https://forms.office.com/r/dTP38LnmsJ
• and last but not lease – 1 50 minutes session on Saturday together with my colleague and friend Bernhard Engleder (t) we’ll talk about Data Governance with Azure Purview – Theory, Customer Insights and Demos. We will share insights from our customer initiatives and how we at cubido guide our customers into the Data Governance journey.

I will be there for the full conference – feel free to stop me and say Hi! I am one of the people with a mask on.. 🙂

More information (schedule) – https://arcade.sqlbits.com/

Registration – https://register.sqlbits.com/event/36db4730-583c-49c1-adcf-f76432bb6580/summary

Stay curious, and more important – stay healthy!

Wolfgang

A new Azure Purview permission level – Policy authors

I’ve spotted a new Azure Purview permission level – Policy Authors. This new permission level is connected to the new Data Access Policy Management in Purview (as of today 2021-12-30 in preview; https://docs.microsoft.com/en-us/azure/purview/how-to-access-policies-storage)

One may ask: Why Wolfgang, why no blog post about that feature? Well, the answer is easy: it’s in preview and the preview limitations are strict – the enforcement side of the policies are limited certain Azure regions. And therefore, I am planning to try it (and blog about it) when it will be available in more regions.

Add a root collection admin to your Azure Purview account (not in Purview Studio)

Some time ago, the permissions scheme in Azure Purview changed and the collection level was introduced as a level of permission. (documentation – Access control in Azure Purview)

With that, the root collection and its assigned administrators get more and more important. The permissions in Azure Purview are configured in Purview Studio (Collections -> select a collection -> Role assignments).

Azure Purview role assignments in Purview studio

What I found out today, is that you can add a Purview Root Collection Admin directly in the Azure Portal. Open you Purview account, and select the Root collection permission menu entry.

Purview Root Collection Admin

Select one member account and add it to the list of collection administrator.

Successfully added an additional root collection admin

Happy permission management,

Wolfgang

Azure Synapse Analytics – Success by Design Documents

Today, I stumbled upon a very interesting link – the Azure Synapse Analytics – Success by Design site (follow this link).

If you need guidance, best practices links, POC playbooks, links to blogs & videos, tools, .. THIS is the site you need to bookmark.

screenshot taken 2021-12-21 (https://azure.github.io/Synapse-Success-By-Design/docs/home/)

One section I really like and definitely will have a closer look at is the Implementation Success. It guides you through the process of setting up the ground for a successful POC project. Even after going through several Synapse POCs & projects, I already spotted some additional things to consider.

The content is provided and maintained by the Azure Synapse Customer Success Engineering (CSE) group.

Additional links:

• Azure Synapse Analytics documentation: https://docs.microsoft.com/en-us/azure/synapse-analytics/
• What’s new in Azure Synapse Analytics? https://docs.microsoft.com/en-us/azure/synapse-analytics/whats-new
• Azure Synapse Analytics Monthly Updates (string October 2021) – https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/bg-p/AzureSynapseAnalyticsBlog/label-name/Monthly%20Update
• Azure Synapse Analytics Blog – https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/bg-p/AzureSynapseAnalyticsBlog

SQLBits 2022 – My Training Day about Data Governance and Azure Purview

I am happy. More than happy that I can share the news now…

Next March, in London, I am allowed to conduct a full day training at SQLBits 2022. About one of my highlight topics in the past months – Azure Purview and Data Governance.

The list of Training days is already published, registration will open early next year. If you are not sure if the content of the workshop titled Data Governance in a Nutshell – From Theory to Hands-On with Azure Purview is suitable for you, here are some details:

Data is important for every organization. To know your data estate (systems), their dependencies, the responsible people and processes takes your company’s data literacy to the next level.
This topic is called Data Governance, which includes two main streams: a) the organizational processes and policies and the b) the technical tools to support the organizational processes.

Many of us think, that Data Governance is a technical problem to solve, but I would start the Data Governance day with an explanation what DG is. We will talk about the people involved, the processes needed, policies that should be in place and the technology so support the processes.
The overall Data Governance vision and strategy will also be discussed.

After the DG theory part, we will dive into a Data Governance tool – Azure Purview. In the Hands-On lab we will connect to data sources, configure scans, see how Purview classifies data, assign and work with a business glossary, work with & analyze data lineage and see the Purview data catalog in action.
In the last part of the day, we will see Purview integration in action – the Apache ATLAS API will be presented as well the integration into Azure Data Factory and Azure Synapse Analytics.

At the end of the day, attendees should have a feeling about what Data Governance is, the organizational requirements as well as a first look into Azure Purview and how a technical answer to these questions can look like.

https://sqlbits.com/information/event22/Data_Governance_in_a_Nutshell__From_Theory_to_Hands-On_with_Azure_Purview/trainingdetails

In short: the day will guide you through the foundation patterns of Data Governance, the way how Azure Purview can solve (some) of the challenges for Data Governance and get you an idea how you and your organization can start/move further on with Data Governance.

The full list of training days can be found there: https://sqlbits.com/information/trainingday

Complete Analytics Fabric – Azure Synapse and Power Query (Power BI) better together

As a follow up of todays PASSDataCommunitySummit keynote, in which Arun Ulag mentioned a new phrase – the Complete Analytics Fabric, I would like to write more about a new option to really get a complete picture of your Azure Synapse Analytics artifacts in your Synapse workspace in one got – in one Power BI dataset – with one connection option only. Sounds interesting? Yes.. it is a nice way..

The new Azure Synapse Analytics Workspace Connector for Power Query

Announced at Ignite 2021 (source: https://techcommunity.microsoft.com/t5/azure-synapse-analytics/supercharge-bi-insights-with-the-new-azure-synapse-analytics/ba-p/2929352) , the new Azure Synapse Analytics Workspace Connector allows you, as a Power Query / Power BI user to directly connect to all the data artefacts in a Synapse workspace. Prior to this new connector, you needed to connect to the SQL Pools and/or Spark database separately.

The November Release of Power BI Desktop includes a preview version of this new connector (Nov 2021 release blog post)

Let’s see how this connector works and which options you get by using it. In my demo example, I am using Power BI Desktop and as a first action, use the Get Data dialog to connect to a Synapse Workspace.

There are (as of today) two options to connect to Synapse Analytics – the “old” Azure Synapse Analytics (SQL DW) connector and the new Synapse Workspace connector. Select the new one and move on to the next step..

The new Azure Synapse Analytics workspace connector

Which is a warning that this is a development connector and third-party software is used to connect. I’ll continue …

The warning when using the connector the first time

Up next – Authenticate using your AAD user that has access to a Synapse Analytics workspace.

The next dialog lists all the Synapse workspace your user has access to

The list of Synapse workspace the authenticated user has access to

Whenever I expand one of the workspace nodes, the available data artifacts (databases) are listed. In my demo workspace, the following objects are available. The question that comes up now is: What are all these objects? SQL Dedicated Pools? Serverless databases? Spark databases?

The data artifacts in my Synapse Workspace

If we compare the list in Power BI Desktop and the objects in Synapse Studio, it will get clearer which objects are now available in the Power Query connection dialog.

Which objects are available in the Power Query connector?

As of today, Spark databases (now called Lake databases in Synapse Studio), SQL Serverless databases and Dedicated Pools are available for a connection.

The Synapse Analytics workspace connector now allows you to select tables (objects) from different sources and them in one connection and data access action.

Select different tables and use them in Power BI

In my example, Power BI did not ask if I want an Import or Direct Query connection – Import mode was chosen for me.

In my opinion, this new connector really makes our life as Synapse and/or Power BI developer easier – only one connector to use them all. Let’s see if the list of supported artifacts is extended (like it is mentioned in the announcement blog post) – this really would be very nice: SQL Pools, Spark databases and data lakes, and even linked services connected in one go.

• Power Query Blog: https://powerquery.microsoft.com/en-us/blog/enable-synapse-workspace-data-in-power-bi/
• Power BI Desktop November 2021 blog post: https://powerbi.microsoft.com/en-us/blog/power-bi-november-2021-feature-summary

Happy data set creation!

How to change URL-copy&paste back to normal (URL only)

Some months ago I switched to Microsoft Edge as my default web browser. I don’t know when it started, but out of a sudden the copy&paste of URLs out of Edge ended strange in the destination application. A new feature in Edge changed the format of copied URLs in a more “user-friendly” format. Which is nice, yes, .. but…

User-Friendly Formatting for copied URLs

Just in case you do not know how this user-friendly URL does look like:

User-friendly URL copy & paste

How to change it back to URL only

For my job, I often have to write documentations and other documents and in those I want to see the real URL. So I started to do a little research and there are several options to the back to the URL only:

• Change the Paste mode in Word (Keep as Text only)

• Change the default way, how Edge copies URL

This is my preferred way – change the default setting how MS Edge copies URLs. Short way: open Edge and enter edge://settings/shareCopyPaste into your address bar (or you open Edge settings and navigate to the Share, copy and paste section).

This opens the settings page for URL sharing. Choose the link option if you want the user-friendly information, choose the plain text version if you want the link only.

I hope you liked this tip – otherwise this post is another reference for future Wolfi 😉

Happy copy&paste,

Wolfgang

PASS Data Community Summit – Keynote

The PASSDataCommunitySummit 2021 kicks off with a Keynote presented by Rohan Kumar

• Limitless cloud database services – choose the way / service that serves you best for your data
• Unmatched analytics and insights – choose the technology / services to analyze your data and get insights out of it
• Unified data governance – Rohan compares it to a shield around your data stores and analytic technologies to get the unified overview.

It’s the #DataCommunity

Community is about working together and for each other..

The #AzureDataCommunity has grown in the past months… >70k members, 130+ user groups around the world…

The Azure Data Communtiy grew within the last months around the globe – >70k members and almost 40 countries

Some events / initiatives from and for the community are presented:

• SQLSaturday
• DataSaturday
• NewStarsOfData

SQL Server 2022 – Preview

The first (early access) private preview is announced – https://cloudblogs.microsoft.com/sqlserver/2021/11/02/announcing-sql-server-2022-preview-azure-enabled-with-continued-performance-and-security-innovation/

Overview about the SQL 2022 features

Bob Ward demos (and Connor joined 🙂 the “it just gets faster without code changes” in SQL Server 2022 (Parameter sensitive plan optimization)

Bob and Connor discussing 🙂

No code change required – only the compatibility level of the database is changed.. and the magic happens

More about the (Azure) SQL Family

The (Azure) SQL family

“Build once – deploy anywhere”

Azure Arc-enabled data services

Azure Cosmos DB

Announcement: Azure Managed Instance for Apache Cassandra

Azure Synapse Link for SQL Server (2022+)

The Azure Synapse Link family is extended to SQL Server (2022+) and will be available in the future. Synapse Link is more than “just a linked server” -> Synapse Link publishes changes to Synapse and can then be used for analytics in Power BI.

Azure Synapse Link for Dataverse

It’s GA. Do some of you remember the Export to DataLake functionality? yes.. Synapse Link for Dataverse is definitely MORE than that…

From Dataverse to Azure Synapse to your the reporting tool of your choice.

Data is pushed from Dataverse to the landing part in Azure Synapse. A Spark database (containing external table definitions to the linked tables) is created. You can use this tables in either Spark or SQL Serverless (using the shared metadata approach) ..

Intelligence Platform

Arun takes over the stage

Empower every individual / every team / every organization

Power BI + Teams integration

Power BI + Excel integration

Power BI Goals

The Complete Analytics Fabric => Power BI and Synapse

Announcements for Power BI

• Patrick talks about Hybrid Tables and Automatic Aggregations.

Analyze the effects of Automatic Aggregations

Hybrid Tables – Import for historical data + Direct Query for the most up-to-date data.

Azure Purview – Data Governance

If you want to learn more about #AzurePurview – there is a video quickstart series available: https://workingondata.wordpress.com/2021/08/31/azure-purview-quickstart-series/

Data Lineage

Data Lineage is one of the key features required for a good understanding of your data landscape. Btw. if you want to learn more about data lineage, g2.com has a very good introduction: https://learn.g2.com/data-lineage

Integration of Purview and Synapse

Integration of Power BI and Purview

Integration of SQL Logic (Stored Procs…)