Why can’t I move my Azure Purview sources between collections?

Yesterday, Andy Cutler (t) asked me on twitter, if the move of a registered source between collections is possible.

Andys question

My first answer was, yes sure… And I also included a screenshot from one of my Purview accounts (which – this will be important in the following – is a legacy Purview account).

Image
Move a source between collections in Azure Purview (legacy Purview account)

But – tech isn’t tech without some differentiation between versions and available options ๐Ÿ˜‰

The new Purview collection management (mentioned here) introduced in August 2021 allows for a finer granular permission definition and other things, but some of the features are still missing: like the move and delete of sources in the data map.

If you are using a new (created after August 18th, 2021) Purview account, the move of sources between collections is not possible (as of today, 2021-09-16)

Sorry @Andy – we’ll hopefully do not have to wait tooo long for the move feature to be available in new Purview accounts.

I would like to say – Happy source-moving.. but we’ll have to wait some time for that,

Wolfgang

Posted in Azure Purview | 1 Comment

Azure Purview: Where are my Assets classified listed?

I got a question from a customer this week, that Azure Purview does not list the Assets classified in the scan history anymore but classifications happen.

Hmm.. What happened?

Remark: All the screenshots were taken in a legacy Purview account (Azure Purview โ€“ Access Control and Collectionย Management).

When I open the scan history list, the list of columns indeed does not contain the Assets classified column.

What is for sure available for a longer period in the UI – the Edit columns action is responsible for the missing columns. This dialog allows you to customize the display columns – et voila – the missing information is listed there.

I also spotted some (for me) new information pieces like Relationships ingested, … which I will for sure have a closer look in the future.

Happy data scanning,

Wolfgang

Posted in Azure Purview, Tips | Leave a comment

Unable to create Azure Purview Account – Resource provider Microsoft.Purview is not registered

In my last Azure Purview Quickstart video (#3 – Create an Azure Purview Account – link), I’ve shown you how to create a new Azure Purview account.

And what pre-prepared demos have in common, well – it “just” works there ๐Ÿ˜‰
BUT: there are some requirements that need to be configured beforehand, in order to create an Azure Purview Account.

Basically, problems during the creation process can be listed to:

  • Security / permissions
  • Missing Resource providers

Permissions required to create an Azure Purview Account

You need Azure Contributor or Owner role (or be an Administrator of the Azure subscription)

Missing Resource providers

Azure Purview requires the following resource providers to be registered in your Azure subscription. If you have not heard of Azure resource providers before, please have a look at the docs page.

Azure Purview requires the resources providers for

  • Microsoft.Purview
  • Microsoft.Storage
  • Microsoft.EventHub

to be activated in your subscription…

The validation error message looks similar to this one:

Validation failed with error: The template deployment ‘Microsoft.AzurePurviewGalleryPackage-datagov01’ is not valid according to the validation procedure. The tracking id is ‘xxxxxxxx’. See inner errors for details.
Detailed error(s):
21005 – The resource providers Microsoft.Storage and Microsoft.EventHub and Microsoft.Purview are not registered for subscription xxxxxx. For improved Azure Policy validation on managed resources, we recommend to register Microsoft.PolicyInsights resource provider on your subscription.

The resource providers Microsoft.Storage and Microsoft.EventHub and Microsoft.Purview are not registered for subscription xxxx

Missing resource providers (Microsoft.Purview) for Purview Account

How to register missing resource providers?

  • First of all – you need administrative access to your Azure subscription to enable resource providers.
  • Open the Azure portal and head over to Subscriptions
Azure subscription list

Select the Resource providers menu entry and search for the three required ones (Microsoft.Purview, Microsoft.Storage, Microsoft.EventHub)

Resource provider menu in the subscription section

(1) Search for the resource provider name, select (2) the entry and (3) register / re-register it in your subscription.

Resource provider details page

Reminder: check for all three resource providers to be registered! ๐Ÿ˜‰

With these configuration steps, you now should be able to create your Azure Purview account!

Happy data cataloguing,

Wolfgang

Posted in Azure, Azure Purview | 2 Comments

Azure Purview Quickstart (Series)

You are new to Azure Purview and want to learn more about this Azure DataGovernance service?
I’ve started a Quickstart Azure Purview – Video Series and would like to invite you to the videos listed below. Feedback and discussion appreciated..

#1 What is Azure Purview?

#2 What is inside? Azure Purview building blocks

#3 – Create an Azure Purview Account (Azure Portal)

Additional information if you get a validation error (resource provider Microsoft.Purview / Microsoft.Storage / Microsoft.EventHub is not registered) -> I’ve collected the required steps to solve this: Unable to create Azure Purview Account โ€“ Resource provider Microsoft.Purview is not registered

#4 Azure Purview Studio

#5 – Azure Purview Data Map, Collections and Register Sources

Posted in Azure, Azure Purview, InformationSharing | 1 Comment

Azure Purview – Recording of my HybridVirtualGroup session – Why and How to Govern your Data Estate with Azure Purview

I am very happy, that the recording of my session for the Hybrid Virtual Group is now published on youtube. Feel free to discuss the topic there..

Thanks to Mark and Mikael for hosting the meeting.

Hybrid Virtual Group

Posted in Azure, Azure Purview, Presentations | Leave a comment

Azure Purview – Enhanced Power BI Integration

It seems like it’s Azure Purview week – another feature enhancement made it to my Purview account. The enhanced metadata scanning for Power BI tenants.

The Power BI integration in Azure Purview was there since the first public preview in December last year. But the assets discovered only got down to the Power BI dataset, report and dashboard level.

With the introduction of the enhanced metadata scanning API in Power BI, well.. more details about your Power BI tenant can be queried through the REST API (announcement blog post). That enhancement to the Power BI APIS now allow Azure Purview to get more granular information out of Power BI.

How do you get this more detailed information? Well – I created a short video for you to demo the new functionality and the steps that are required to get Azure Purview talk well to Power BI.

Azure Purview and Power BI Integration

Configuration Steps:

  • Azure Active Directory
    • Create a Security Group and add the Azure Purview Service Principal to that group
  • Power BI Admin portal
    • Allow the Purview service principal to access the Power BI API (you’ll use the created security group)
    • Enable – Allow service principals to use read-only Power BI admin APIs, and Enhance admin APIs responses with detailed metadata
Configuration settings in the Power BI Admin portal – allow the security group (for Azure Purview) to use the Power BI metadata APIs
  • Azure Purview
    • Configure a Power BI source
    • Configure a Scan
    • Browse your Power BI assets

What is included in the (enhanced) Power BI metadata?

What I found out so far – For Power BI datasets, there are more properties extracted now:

Dataset properties including the “hierarchy” -> in this screenshot the dataset is published into a shared capacity (Power BI PRO)

Power BI dataset properties in Azure Purview
Properties part 2

This is the really new part – Power BI dataset schema information.

Initially during my first tests, the schema part of the Purview asset did not populate. Together with the Purview team, we could solve this.

๐ŸŽฏIf no Power BI schema information is extracted into Purview, make sure that the Power BI dataset is at least – one time refreshed in the service. I uploaded my demo dataset without a refresh in the service afterwards. The refresh (and a following Purview scan) solved the population of the asset section.

What is really new: Schema information

You can even drill down to a specific column / measure of the dataset

What is missing (in August 2021 and for Wolfi ๐Ÿ™‚ ?

Well, there is always something that is missing.. ๐Ÿ™‚ What I found as a start:

  • Data Lineage is still available on the data set / report / dashboard level -> detailed (column level) lineage would be really, really nice… (nicely looking into the direction of the Purview product team ๐Ÿ™‚
  • What would help, maybe, would be the DAX formula for a Measure. There is already a flag isMeasure set to true, but the DAX formula is not extracted

How do you like the enhanced Power BI & Azure Purview integration? What is missing for you? Is it working for yourself?

Happy data cataloging,

Wolfgang

Posted in Azure, Azure Purview, PowerBI | Leave a comment

Azure Purview – Access Control and Collection Management

In the past days, it looked like Azure Purview gets new features every day, or multiple new features per day. Today, I spotted the new Catalog Permissions section in the Purview docs. Some of you may think.. well, access control does not sound THAT interesting.. but this one for sure is – You now can configure your access on the Purview collection level. No more – one-role-to-rule-them all.. permissions are now getting more granular in Purview.

For me, this really looked very interesting and I had to try it out .. and today, something new for you my readers – I recorded a short video.

Azure Purview access control – the new stuff from August 2021

As it was a spontaneous idea, please provide me with feedback.

Some things to consider for the Access Control topic:

  • You need to create a new Purview account (after August 18th 2021) to get the new collection experience. The docs do not mention an upgrade option.
  • I did not found a way to delete a sub-collection -> this was possible in the legacy collection experience
  • There is (currently) no way to rename a sub-collection -> be aware and choose your names wisely

Stay curious, catalog your data and #AzureData rocks,

Wolfgang

Posted in Azure, Azure Purview | 2 Comments

Azure Purview – Set a Default Account

Ha – something new in the Azure Portal ๐Ÿ˜‰

When I had a look a my Purview accounts, I recognized a change in the list view for Purview accounts – View default account

Purview accounts lists

And curious Wolfi thought.. what’s that? And well, the answer is easy – you now can define a default purview account and with that shortcut button, it’s easy to open this one

If no default account is defined..

No Default Purview account set

If you open the details of a Purview account (NOT Purview Studio) in the Azure Portal, there is a link in the Essentials section that allows you to set the default entry. And that’s it.

Set a default Azure Purview account

If a default account is set, the View default account link opens the following dialog

Default Purview account is defined

An easy, but handy feature…

Wolfgang

Posted in Azure, Azure Purview, Best Practices | Leave a comment

Azure Purview Elastic Data Map

It’s been a long and intensive discussion – the (initial) pricing structure of Azure Purview. As I already talked about it in my Purview pricing blog post, the basic cost calculation involves

  • the cost for the data map (= the infrastructure to store metadata and provide the Purview UI + cataloging functionality)
  • plus the costs involved for scanning sources.

The data map costs were fixed (until today) to either 4 capacity units(CU) or 16 capacity units. Without an option to stop / pause / react to minimal performance needs ….

Purview Elastic Data Map

Today (2021-08-18), the Azure Purview team announced the Elastic data map concept (https://docs.microsoft.com/en-us/azure/purview/concept-elastic-data-map) which allows for an auto-scaling of the underlying infrastructure. The minimum data map size is now 1 CU with is named a data map capacity unit (DMCU ;-)) which equals to a throughput of 25 operations/sec and 2 GB of metadata storage.

  • The data map is auto scaled – which means if you / Azure Purview exceeds the current performance ranges (operations/sec or metadata storage), the data map is scaled.
  • The data map is billed on an hourly basis – the maximum peak defines the number of DMCU you are charged.
  • The data map is scaled up and down between the defined quotas (https://docs.microsoft.com/en-us/azure/purview/how-to-manage-quotas)

What about my existing Purview account?

According to the email sent from Azure Purview team, existing Purview accounts will be transferred to an underlying elastic data map in the September/October timeframe.

What about new Purview accounts?

As of today, new Purview accounts are created with an elastic data map beneath – the Create a Purview Account dialog already shows this:

Create A new Purview Account dialog in the Azure portal

And now – the hard facts… costs..

Well.. save the best for the last.. they said..

And to be honest – I cannot name a price because.. the pricing page (https://azure.microsoft.com/en-us/pricing/details/azure-purview/#overview) is updated and mentions this:

What is really interesting – what will be the price for a CU / hour? plus .. it’s mentioning: Capacity Units free in Preview.. which would in turn mean that there should be no charges in this timeframe for the data map.

We’ll see..

Stay curious, catalog & govern your data!

Wolfgang

Posted in Azure, Azure Purview | 1 Comment

Free Azure Data Summer: Architecture Guides

Hi and welcome back to another edition of the #FreeAzureDataSummer!

Today, I would like to bring your attention to a – for me – very interesting and useful link – the Azure Application Architecture Guide. If you open the browser of your choice and head over to https://docs.microsoft.com/en-us/azure/architecture/guide/, you’ll find a huge list of possible architectures around different Azure solution cases.

Ranging from design principles, technology choices and best practices advices, the list is a long one – take your time and browse through the available list of architectural diagrams.

What I’ve learned in my day-to-day project work is that there is no – one-size or one-solution-fits-all approach. The Azure Architecture Guide is a good starting point (and not even that – a good reference point) for getting the best possible solution for your current use case.

With that, enjoy the day and find the right architecture for your project ๐Ÿ˜‰

Wolfgang

link to the Azure Architecture Guide:

Posted in Azure, FreeAzureDataSummer, InformationSharing | Leave a comment