Data News – 2021-02-22

It’s Monday again and it’s time for #WolfisDataNews again!

Power BI

Analytics

(Virtual) Events in the next weeks

Reading is fun, but (virtual) conferences too…

  • Microsoft Ignite – March 2-4 2021https://myignite.microsoft.com/home
  • On-Demand sessions are available: Around the Clock – Azure SQL and Azure Data Factory https://github.com/microsoft/aroundtheclock
  • 18. March 2021 – Join us for the Azure Synapse Launch event – Microsoft Austria and cubido provide you with a webinar full of Synapse and the future of Analytics. Save the date – The registration page will open soon.

Have a good start into the week!

Wolfgang

Posted in Azure Purview, Azure Synapse Analytics, DataNews, InformationSharing, Power Platform, PowerBI | Leave a comment

Data News – 2021-02-15

It’s Monday again and it’s time for #WolfisDataNews again!

Power BI

Analytics

(Virtual) Events in the next weeks

Reading is fun, but (virtual) conferences too…

Have a good start into the week!

Wolfgang

Posted in Azure Purview, Azure Synapse Analytics, DataNews, InformationSharing, Power Platform, PowerBI | Leave a comment

Data News – 2021-02-08

Another week and other data links for you! I hope you had a great weekend – we had spring temperature for a few days in Austria – some gardening plus a long run (21k) .. and now some data news… what a great weekend!

Power BI

Azure Data / Azure Purview & Synapse

(Virtual) Events in the next weeks

Reading is fun, but (virtual) conferences too…

Have a good start into the week!

Wolfgang

Posted in Azure Purview, Azure Synapse Analytics, DataNews, InformationSharing, Power Platform, PowerBI | Leave a comment

Data Types Conversion Matrix in Power Query

Another useful page for my future reference – Data Types in Power Query describes the available datatypes PLUS a very, very useful Data Types conversion matrix.

Convert a decimal number to a date? yes, no, maybe, with a loss of information?

Have a look here and you know if it / and how it works!

Power Query data type conversion map (screenshot taken 2021-02-07, Data types in Power Query | Microsoft Docs)

Happy to help,

Wolfgang

Posted in Best Practices, Business Intelligence, InformationSharing, PowerBI | 1 Comment

Azure Purview Preview (Free) Pricing – Check your Azure Subscription type!

Updates to this topic:

Chapters added at the end of the blog post

  • 2021-02-27: Clarification of billing of Azure Purview
  • 2021-02-27: Extension of the Azure Purview Free Preview Period

The initial blog post

Beginning of December 2020, Azure Purview was released into preview. During the announcement, Microsoft stated that Purview will be free for a certain period of time – depending on the system part of Purview.

DateTime of screenshots taken: 2021-01-29 9pm CET!

Azure Purview Data Map requires capacity units to work and the pricing page mentions that 4 capacity units are free until February 28, 2021.

Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)

The Scanning and Classification section of the pricing page states that scanning Power BI online and SQL Server on-premises is free in preview, whereas other data sources are charged by vCore Hours.

Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)

The Purview Data Catalog part mentions free tiers for level C1 and D0 during preview.

Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)

Attention! Check your Azure Billing and subscription type

Unfortunately the free pricing during the Purview preview / until the end of February 2021 is not available for all kinds of Azure Subscriptions. I found out by accident because my Azure Sponsorship Subscription balance went down with a significant high number of Euros during the last week. 😦

And the reason is, that Azure Purview Data Map (those 4 required capacity units) are NOT FREE in my sponsorship subscription. To additionally increase the bill, I created two Purview instances in my subscription …. hmm.. double the costs..

Let’s do the Purview Maths

Now let’s do the math here:

  • €0.289 per 1 Capacity Unit Hour
  • The minimum size of a Purview Data Map instance is 4 Capacity Units
  • € 0.289 x 4 capacity units x 24 hours = 27.74 € per DAY!
  • € 27.74 x 31 days = 860 € per month
  • € 27.74 x 365 days = 10126.56 € a year

only for the capacity units required for the smallest Data Map instance!!! No scanning costs included!!

WOW – let this sink…

To be honest, I have not really looked in detail at the Purview pricing so far but the overall costs really need to be looked at. Whenever it is during a POC (because no free testing option is available) AND especially when planning and implementing a Data Governance project.
As of today, the Azure Purview Data Map is a 24×7 service – no option to pause the service. Which means – please check your Azure subscription type during the preview period!

Why?

On 2021-02-01, the Purview pricing page (https://azure.microsoft.com/en-us/pricing/details/azure-purview/) was updated:

It now includes the updated eligible Azure subscription types:

Azure Purview Data Map: The 4 free capacity units are only available for customers on the Pay-As-You-Go (MS-AZR-0003P), Microsoft Azure Enterprise (MS-AZR-0017P), Microsoft Azure Plan (MS-AZR-0017G), Azure in CSP (MS-AZR-0145P), and Enterprise Dev/Test (MS-AZR-0148P) offer types. Free quantities are applied at the enrollment level for enterprise customers. Free quantities are applied at the subscription level for pay-as-you-go customers.

Scanning and Classification: The 16 vCore-hours of free scanning are only available for customers on the Pay-As-You-Go (MS-AZR-0003P), Microsoft Azure Enterprise (MS-AZR-0017P), Microsoft Azure Plan (MS-AZR-0017G), Azure in CSP (MS-AZR-0145P), and Enterprise Dev/Test (MS-AZR-0148P) offer types. Free quantities are applied at the enrollment level for enterprise customers. Free quantities are applied at the subscription level for pay-as-you-go customers. Note: Azure Purview provisions a storage account and an Azure Event Hubs account as managed resources. This may incur separate charges that in most cases will not exceed 2% of charges for scanning. Refer to the Managed Resources section in the Azure portal within Azure Purview Resource JSON.

Pricing information for Azure Purview (screenshot taken 2021-02-01 9am CET)

So now – What is my subscription type? Where do I find this information?

You need the appropriate rights to view the subscription details in your Azure Portal. Open the Azure Portal (https://portal.azure.com) and open your list of Subscriptions and head over to the Subscription details.

In one of my Azure Sponsorship subscription, the Offer ID (subscription type) is MS-AZR-0036P = not eligible for the free Purview preview.

The Azure documentation page Understand Cost Management data (link) contains the list of available Azure offers and subscription types.

Updates to this blog post:

2021-02-27: Clarification of billing of Azure Purview

On Tuesday 23rd, February 2021 I got an email titled with Clarification of Billing for Azure Purview and free quantities in January and February 2021.

The email contains information about the fact that the free capacity units costs during the preview time were sometimes charged incorrectly. AND: Microsoft will refund these incorrectly billed charges back. What a nice surprise!

Refunding information

2021-02-27: Extension of the Azure Purview Free Preview Period

Another nice surprise went live on Saturday 27th February. The Purview team tweeted (link to the tweet) that the 4 Capacity Units of the Data Map will stay free until May 31, 2021!
Thank you Purview Team!

To conclude

  • Get to know your Azure subscription type & check your Azure credits balance!
  • Try Purview!

Stay curious,

Wolfgang

Posted in Azure Purview, Data Governance, InformationSharing | 1 Comment

Data News – 2021-02-01

Another week and other data links for you! I hope you had a great weekend – I spent mine outdoors – long walks and long runs.. and it was great to be outside in the snow!

Power BI

Azure Data / Azure Purview & Synapse

(Virtual) Events in the next weeks

Reading is fun, but (virtual) conferences too…

Have a good start into the week!

Wolfgang

Posted in Azure Purview, Azure Synapse Analytics, DataNews, InformationSharing, Power Platform, PowerBI | Leave a comment

The future of Power BI (in the next six months aka 2021 release wave 1 plan)

Yesterday, the new Power Platform release wave was released. The 2021 release wave 1 plan (puh πŸ™‚ ) includes the features planned for April 2021 through September 2021.

If you are curious, what the next months will bring into the Power BI universe, this page is the right one: https://aka.ms/PowerPlatformWave1ReleasePlan

2021 Release Wave 1 Plan

In personally prefer the web page since it is easier to browse through the different chapters and features, but there is also a PDF file available for download.

What is planned for Power BI?

The list of new features is long, therefore I will not repeat and copy it from the release plan itself. But I would like to highlight some of the features:

In the Premium area, Workspace level refresh for data sets and dataflows, dataflow support in Deployment pipelines, the paginated report visual for Power BI and the Performance Accelerator for Azure Synapse Analytics caught my attention.

Next, the enhancements to the lineage, impact and APIs are things to wait for and background validation for dataflows.

Power Query news are somehow hidden in the Data Integration part of the release plan. I personally look forward to the Visual data prep (Diagram View) in Power Query Desktop, Personal custom function library, Gateway automation and management enhancements (diagnostics, export gateway details, security, ..)

This list is only a small excerpt of the huge list of features planned. Have a look at the web page and re-visit it from time to time. There are updates happening from time to time!

Posted in Power Platform, PowerBI | 2 Comments

Better Together – Azure Synapse Analytics and Azure Purview

In the past months I had the chance to play with and build solutions based on Azure Synapse Analytics and Azure Purview.

Azure Synapse (my Synapse blog entries) as the foundation for a solid platform to store, analyze and build data solutions and Azure Purview (my Purview blog posts) as the data governance and data catalog solution in Azure.

During the writing of my latest blog post (What’s new in Azure Synapse Analytics?), I found a very interesting entry in the update feature list: Azure Purview Integration.

Azure Purview Integration in Azure Synapse is announced

And this really got my attention – use Azure Purview to find your data artefacts and use it for exploration… sounds nice.. let’s try it!

Configuration – Connect Azure Synapse to Azure Purview

In the Manage Hub of Azure Synapse, the new menu entry Azure Purview (Preview) is the entry point to connect your two Azure services.

Azure Purview integration in Azure Synapse

You need Contributor rights in Azure Synapse and access to your Purview instance. In the connection dialog, use the browse option or enter the Purview ID into the manual section.

Connect to a Purview account

When connected, the config section allows you to edit the mapping (i.e. change it to another purview instance), refresh (I am not sure what this action really does, but yes, refresh the connection) or disconnect the mapping. With the link on the purview instances name you’ll directed to the Purview studio.

Connection successful – My Purview account is connected to my Azure Synapse Workspace

Use the Purview integration in Azure Synapse

According to the documentation, you can use the Purview integration to discover data registered and scanned by Azure Purview

This integration search is implemented the different hubs / development areas in Synapse Studio. It is available in the Data, Develop and Integrate hub in Synapse studio. After the Synapse / Purview connection is configured, a click into the search box allows you to select the search scope – let it be Purview or the (Synapse) Workspace.

Search in Azure Synapse with Purview connection enabled

Let’s search for the term Customer in our Purview account. The results are listed in the Purview search tab which is similar to the search results tab directly built-in Purview Studio.

Result list – Purview entities in Azure Synapse

For the demo of the functionality, I first open the Customer table (in an Azure SQL database). The detail page for a table in an Azure SQL database looks very familiar to the one in Purview directly. Purview features to browse your data assets are there as well. If your Purview credentials allow, you are also able to edit the properties of the entity (see my comments about security later in this blog post).

Start with an Azure SQL table

Detail page of an Azure SQL table in Azure Synapse (using Purview integration)

With two differences….There is a Connect and a Develop menu. Remember, we are located on the Azure SQL table detail page. With the Purview integration

Menu actions available for an Azure SQL table

As you can see in the screenshot, using the Connect menu allows you to create a linked service definition or a New integration dataset. For the definition of a new integration dataset, you’ll need to create the linked service first. At the moment (2021-01-27) and according to the documentation, the Synapse/Purview integration is not able to infer if there is an existing linked service or integration dataset.

Options in the Connect menu in Azure Synapse/Purview integration

In the Develop section, with an Azure SQL table selected, we’ve got the option to create a New data flow.

Azure Purview integration in Synapse -> directly create a new data flow based on an Azure SQL table entity in Azure Purview.

Or with a file in a Data Lake

Switching back to the results list and next – open the wwi-dimcustomer.csv file in the ADLS Gen2 account. The overall detail page looks very similar to the one of the Azure SQL table, the Develop menu contains some more actions.

Details page for a file in the data lake (Azure Synapse / Purview integration)

For the selected entity (file in the data lake), the Develop menu allows you to directly start with your data exploration either using SQL OPENROWSET options, integrated Spark or a new data flow.

Develop options for a file in the data lake (Purview entity – Azure Synapse integration)

How about the credentials / security?

Based on the documentation page, the Purview integration in Synapse requires you to have Purview access. Synapse passes-through your Purview permissions.

To read more about Azure Purview roles (and their permissions in the catalog), I recommend you to have a look at the documentation.

To sum it up.. Better together? yes? no? YES!

In my opinion, the Purview integration in Azure Synapse is great step into the right direction.

  • With Synapse positioned as the central analytics platform, the direct access to Data Catalog information without leaving Synapse Studio is great.
  • Next, the immediate creation of linked services, integration data sets, or data exploration at your context menu is really a this makes it really easy to integrate data into Azure Synapse.

Stay curious, catalog your data artefacts AND enjoy your day! πŸ˜‰

Wolfgang

Posted in Azure, Azure Purview, Azure Synapse Analytics, Data Governance, Data Lake, InformationSharing | 1 Comment

What’s new in Azure Synapse Analytics?

Just a quick tip today:

The pace of innovation in a cloud environment is a high one. Being updated almost continuously, it is somehow hard to follow all the announcements.

In Azure Synapse, there is one menu entry that informs us about new features, updates and lists the release notes. The update icon is available in the main menu and highlighted in the next screenshot.

The updates menu in Azure Synapse Analytics

Whenever you open the updates menu, you’ll get to a list of the latest updates.

The list of latest updates in Azure Synapse Analytics

Stay curious,

Wolfgang

Posted in Azure, Azure Synapse Analytics, InformationSharing | 1 Comment

Data News – 2021-01-25

Welcome to 2021! Herzlich Willkommen in 2021!

Another week and other data links for you! I hope you had a great weekend – I spent mine outdoors – long walks and long runs.. and it was great to be outside in the snow!

But now – let’s come back to dataplatform news.

Azure Synapse

Power BI

We Don’t Need Data Scientists, We Need Data Engineers! What do you think? Is this a right assumption? We Don’t Need Data Scientists, We Need Data Engineers – Mihail Eric

(Virtual) Events in the next weeks

Reading is fun, but (virtual) conferences too…

Have a good start into the week!

Wolfgang

Posted in Azure Synapse Analytics, Cloud, Conference, DataNews, InformationSharing, PowerBI | Leave a comment