Azure Purview – Azure Data Governance Event

Today, the Azure Data Governance event happened. As you might have noticed, I’ve focused on Azure Purview in the last few months and I really looked forward to this event..

So what happened? What was announced?

Rohan Kumar started the event and talked about the importance of data in enterprises and how Data Governance can help to increase the value of data.

Purview is GA (General Available)

I kind of expected it – Azure Purview is General Available as of today.

Read more about it in the announcement blog post: https://azure.microsoft.com/en-us/blog/govern-your-data-wherever-it-resides-with-azure-purview/

The Azure Purview platform

Afterwards, Mike Flasko took over and went into the more detailed part of the presentation focusing on the Azure Purview platform

Purview Demo

The Azure Purview demo was presented by Evangeline White from the Azure Data Governance team. She showed some interesting stuff like – SQL stored procedure support and the enhanced metadata lineage extraction for Power BI.

Feature summary

The feature overview concluded the event:

But what about the future of Azure Purview / Azure Data Governance?

There was one slide shown, that included some interesting roadmap steps ahead of us..

And hey Wolfi – What about the costs?

Well, the discussion of Purview costs is a long story (blog post) but the pricing page is now updated and there is still some time for you to test it for free (until Nov 1st, 2021)

Some links:

• Azure Purview documentation: https://docs.microsoft.com/en-us/azure/purview
• The Purview GA blog post: https://azure.microsoft.com/en-us/blog/govern-your-data-wherever-it-resides-with-azure-purview/
• Azure Purview pricing: https://azure.microsoft.com/en-us/pricing/details/azure-purview/
• Some Purview blog post: https://workingondata.wordpress.com/category/azure-purview/
• You want to start with Purview today? Have a look at my Purview Quickstart video series… https://workingondata.wordpress.com/2021/08/31/azure-purview-quickstart-series/

Why can’t I move my Azure Purview sources between collections?

Yesterday, Andy Cutler (t) asked me on twitter, if the move of a registered source between collections is possible.

Andys question

My first answer was, yes sure… And I also included a screenshot from one of my Purview accounts (which – this will be important in the following – is a legacy Purview account).

Move a source between collections in Azure Purview (legacy Purview account)

But – tech isn’t tech without some differentiation between versions and available options 😉

The new Purview collection management (mentioned here) introduced in August 2021 allows for a finer granular permission definition and other things, but some of the features are still missing: like the move and delete of sources in the data map.

If you are using a new (created after August 18th, 2021) Purview account, the move of sources between collections is not possible (as of today, 2021-09-16)

Sorry @Andy – we’ll hopefully do not have to wait tooo long for the move feature to be available in new Purview accounts.

I would like to say – Happy source-moving.. but we’ll have to wait some time for that,

Wolfgang

Azure Purview: Where are my Assets classified listed?

I got a question from a customer this week, that Azure Purview does not list the Assets classified in the scan history anymore but classifications happen.

Hmm.. What happened?

Remark: All the screenshots were taken in a legacy Purview account (Azure Purview – Access Control and Collection Management).

When I open the scan history list, the list of columns indeed does not contain the Assets classified column.

What is for sure available for a longer period in the UI – the Edit columns action is responsible for the missing columns. This dialog allows you to customize the display columns – et voila – the missing information is listed there.

I also spotted some (for me) new information pieces like Relationships ingested, … which I will for sure have a closer look in the future.

Happy data scanning,

Wolfgang

Unable to create Azure Purview Account – Resource provider Microsoft.Purview is not registered

In my last Azure Purview Quickstart video (#3 – Create an Azure Purview Account – link), I’ve shown you how to create a new Azure Purview account.

And what pre-prepared demos have in common, well – it “just” works there 😉
BUT: there are some requirements that need to be configured beforehand, in order to create an Azure Purview Account.

Basically, problems during the creation process can be listed to:

• Security / permissions
• Missing Resource providers

Permissions required to create an Azure Purview Account

You need Azure Contributor or Owner role (or be an Administrator of the Azure subscription)

Missing Resource providers

Azure Purview requires the following resource providers to be registered in your Azure subscription. If you have not heard of Azure resource providers before, please have a look at the docs page.

Azure Purview requires the resources providers for

• Microsoft.Purview
• Microsoft.Storage
• Microsoft.EventHub

to be activated in your subscription…

The validation error message looks similar to this one:

Validation failed with error: The template deployment ‘Microsoft.AzurePurviewGalleryPackage-datagov01’ is not valid according to the validation procedure. The tracking id is ‘xxxxxxxx’. See inner errors for details.
Detailed error(s):
21005 – The resource providers Microsoft.Storage and Microsoft.EventHub and Microsoft.Purview are not registered for subscription xxxxxx. For improved Azure Policy validation on managed resources, we recommend to register Microsoft.PolicyInsights resource provider on your subscription.

The resource providers Microsoft.Storage and Microsoft.EventHub and Microsoft.Purview are not registered for subscription xxxx

Missing resource providers (Microsoft.Purview) for Purview Account

How to register missing resource providers?

• First of all – you need administrative access to your Azure subscription to enable resource providers.

• Open the Azure portal and head over to Subscriptions

Azure subscription list

Select the Resource providers menu entry and search for the three required ones (Microsoft.Purview, Microsoft.Storage, Microsoft.EventHub)

Resource provider menu in the subscription section

(1) Search for the resource provider name, select (2) the entry and (3) register / re-register it in your subscription.

Resource provider details page

Reminder: check for all three resource providers to be registered! 😉

With these configuration steps, you now should be able to create your Azure Purview account!

Happy data cataloguing,

Wolfgang

Azure Purview Quickstart (Series)

You are new to Azure Purview and want to learn more about this Azure DataGovernance service?
I’ve started a Quickstart Azure Purview – Video Series and would like to invite you to the videos listed below. Feedback and discussion appreciated..

#1 What is Azure Purview?

#2 What is inside? Azure Purview building blocks

#3 – Create an Azure Purview Account (Azure Portal)

Additional information if you get a validation error (resource provider Microsoft.Purview / Microsoft.Storage / Microsoft.EventHub is not registered) -> I’ve collected the required steps to solve this: Unable to create Azure Purview Account – Resource provider Microsoft.Purview is not registered

#4 Azure Purview Studio

#5 – Azure Purview Data Map, Collections and Register Sources

#6 – Scan Definitions and Scan Rule Sets (or how to get metadata into the Data Map)

#7 – Browse your Data Assets and get a Data Catalog Intro

Azure Purview – Recording of my HybridVirtualGroup session – Why and How to Govern your Data Estate with Azure Purview

I am very happy, that the recording of my session for the Hybrid Virtual Group is now published on youtube. Feel free to discuss the topic there..

Thanks to Mark and Mikael for hosting the meeting.

Hybrid Virtual Group

• Meetup: https://www.meetup.com/en-us/hybrid-virtual-group/
• Linkedin: https://www.linkedin.com/groups/12487962/

Azure Purview – Enhanced Power BI Integration

It seems like it’s Azure Purview week – another feature enhancement made it to my Purview account. The enhanced metadata scanning for Power BI tenants.

The Power BI integration in Azure Purview was there since the first public preview in December last year. But the assets discovered only got down to the Power BI dataset, report and dashboard level.

With the introduction of the enhanced metadata scanning API in Power BI, well.. more details about your Power BI tenant can be queried through the REST API (announcement blog post). That enhancement to the Power BI APIS now allow Azure Purview to get more granular information out of Power BI.

How do you get this more detailed information? Well – I created a short video for you to demo the new functionality and the steps that are required to get Azure Purview talk well to Power BI.

Azure Purview and Power BI Integration

Configuration Steps:

• Azure Active Directory
  • Create a Security Group and add the Azure Purview Service Principal to that group
• Power BI Admin portal
  • Allow the Purview service principal to access the Power BI API (you’ll use the created security group)
  • Enable – Allow service principals to use read-only Power BI admin APIs, and Enhance admin APIs responses with detailed metadata

Configuration settings in the Power BI Admin portal – allow the security group (for Azure Purview) to use the Power BI metadata APIs

• Azure Purview
  • Configure a Power BI source
  • Configure a Scan
  • Browse your Power BI assets

What is included in the (enhanced) Power BI metadata?

What I found out so far – For Power BI datasets, there are more properties extracted now:

Dataset properties including the “hierarchy” -> in this screenshot the dataset is published into a shared capacity (Power BI PRO)

Power BI dataset properties in Azure Purview

Properties part 2

This is the really new part – Power BI dataset schema information.

Initially during my first tests, the schema part of the Purview asset did not populate. Together with the Purview team, we could solve this.

🎯If no Power BI schema information is extracted into Purview, make sure that the Power BI dataset is at least – one time refreshed in the service. I uploaded my demo dataset without a refresh in the service afterwards. The refresh (and a following Purview scan) solved the population of the asset section.

What is really new: Schema information

You can even drill down to a specific column / measure of the dataset

What is missing (in August 2021 and for Wolfi 🙂 ?

Well, there is always something that is missing.. 🙂 What I found as a start:

• Data Lineage is still available on the data set / report / dashboard level -> detailed (column level) lineage would be really, really nice… (nicely looking into the direction of the Purview product team 🙂
• What would help, maybe, would be the DAX formula for a Measure. There is already a flag isMeasure set to true, but the DAX formula is not extracted

How do you like the enhanced Power BI & Azure Purview integration? What is missing for you? Is it working for yourself?

Happy data cataloging,

Wolfgang

Azure Purview – Access Control and Collection Management

In the past days, it looked like Azure Purview gets new features every day, or multiple new features per day. Today, I spotted the new Catalog Permissions section in the Purview docs. Some of you may think.. well, access control does not sound THAT interesting.. but this one for sure is – You now can configure your access on the Purview collection level. No more – one-role-to-rule-them all.. permissions are now getting more granular in Purview.

For me, this really looked very interesting and I had to try it out .. and today, something new for you my readers – I recorded a short video.

Azure Purview access control – the new stuff from August 2021

As it was a spontaneous idea, please provide me with feedback.

Some things to consider for the Access Control topic:

• You need to create a new Purview account (after August 18th 2021) to get the new collection experience. The docs do not mention an upgrade option.
• I did not found a way to delete a sub-collection -> this was possible in the legacy collection experience
• There is (currently) no way to rename a sub-collection -> be aware and choose your names wisely

Stay curious, catalog your data and #AzureData rocks,

Wolfgang

Azure Purview – Set a Default Account

Ha – something new in the Azure Portal 😉

When I had a look a my Purview accounts, I recognized a change in the list view for Purview accounts – View default account…

Purview accounts lists

And curious Wolfi thought.. what’s that? And well, the answer is easy – you now can define a default purview account and with that shortcut button, it’s easy to open this one

If no default account is defined..

No Default Purview account set

If you open the details of a Purview account (NOT Purview Studio) in the Azure Portal, there is a link in the Essentials section that allows you to set the default entry. And that’s it.

Set a default Azure Purview account

If a default account is set, the View default account link opens the following dialog

Default Purview account is defined

An easy, but handy feature…

Wolfgang

Azure Purview Elastic Data Map

It’s been a long and intensive discussion – the (initial) pricing structure of Azure Purview. As I already talked about it in my Purview pricing blog post, the basic cost calculation involves

• the cost for the data map (= the infrastructure to store metadata and provide the Purview UI + cataloging functionality)
• plus the costs involved for scanning sources.

The data map costs were fixed (until today) to either 4 capacity units(CU) or 16 capacity units. Without an option to stop / pause / react to minimal performance needs ….

Purview Elastic Data Map

Today (2021-08-18), the Azure Purview team announced the Elastic data map concept (https://docs.microsoft.com/en-us/azure/purview/concept-elastic-data-map) which allows for an auto-scaling of the underlying infrastructure. The minimum data map size is now 1 CU with is named a data map capacity unit (DMCU ;-)) which equals to a throughput of 25 operations/sec and 2 GB of metadata storage.

• The data map is auto scaled – which means if you / Azure Purview exceeds the current performance ranges (operations/sec or metadata storage), the data map is scaled.
• The data map is billed on an hourly basis – the maximum peak defines the number of DMCU you are charged.
• The data map is scaled up and down between the defined quotas (https://docs.microsoft.com/en-us/azure/purview/how-to-manage-quotas)

What about my existing Purview account?

According to the email sent from Azure Purview team, existing Purview accounts will be transferred to an underlying elastic data map in the September/October timeframe.

What about new Purview accounts?

As of today, new Purview accounts are created with an elastic data map beneath – the Create a Purview Account dialog already shows this:

Create A new Purview Account dialog in the Azure portal

And now – the hard facts… costs..

Well.. save the best for the last.. they said..

And to be honest – I cannot name a price because.. the pricing page (https://azure.microsoft.com/en-us/pricing/details/azure-purview/#overview) is updated and mentions this:

What is really interesting – what will be the price for a CU / hour? plus .. it’s mentioning: Capacity Units free in Preview.. which would in turn mean that there should be no charges in this timeframe for the data map.

We’ll see..

Stay curious, catalog & govern your data!

Wolfgang