18. March 2021 – Join us for the Azure Synapse Launch event – Microsoft Austria and cubido provide you with a webinar full of Synapse and the future of Analytics. Save the date – The registration page will open soon.
Python – What is Python? How and where to start? If you are new to Python and want to improve? have a look at the 44 part free video training – Programming for Python | Python for Beginners – https://www.youtube.com/watch?v=jFCNu1-Xdsw&feature=youtu.be
Another week and other data links for you! I hope you had a great weekend – we had spring temperature for a few days in Austria – some gardening plus a long run (21k) .. and now some data news… what a great weekend!
Another useful page for my future reference – Data Types in Power Query describes the available datatypes PLUS a very, very useful Data Types conversion matrix.
Convert a decimal number to a date? yes, no, maybe, with a loss of information?
Have a look here and you know if it / and how it works!
2021-02-27: Clarification of billing of Azure Purview
2021-02-27: Extension of the Azure Purview Free Preview Period
The initial blog post
Beginning of December 2020, Azure Purview was released into preview. During the announcement, Microsoft stated that Purview will be free for a certain period of time – depending on the system part of Purview.
DateTime of screenshots taken: 2021-01-29 9pm CET!
Azure Purview Data Map requires capacity units to work and the pricing page mentions that 4 capacity units are free until February 28, 2021.
Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)
The Scanning and Classification section of the pricing page states that scanning Power BI online and SQL Server on-premises is free in preview, whereas other data sources are charged by vCore Hours.
Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)
The Purview Data Catalog part mentions free tiers for level C1 and D0 during preview.
Pricing information for Azure Purview (screenshot taken 2021-01-29 9pm CET)
Attention! Check your Azure Billing and subscription type
Unfortunately the free pricing during the Purview preview / until the end of February 2021 is not available for all kinds of Azure Subscriptions. I found out by accident because my Azure Sponsorship Subscription balance went down with a significant high number of Euros during the last week. 😦
And the reason is, that Azure Purview Data Map (those 4 required capacity units) are NOT FREE in my sponsorship subscription. To additionally increase the bill, I created two Purview instances in my subscription …. hmm.. double the costs..
Let’s do the Purview Maths
Now let’s do the math here:
€0.289 per 1 Capacity Unit Hour
The minimum size of a Purview Data Map instance is 4 Capacity Units
€ 0.289 x 4 capacity units x 24 hours = 27.74 € per DAY!
€ 27.74 x 31 days = 860 € per month
€ 27.74 x 365 days = 10126.56 € a year
only for the capacity units required for the smallest Data Map instance!!! No scanning costs included!!
WOW – let this sink…
To be honest, I have not really looked in detail at the Purview pricing so far but the overall costs really need to be looked at. Whenever it is during a POC (because no free testing option is available) AND especially when planning and implementing a Data Governance project. As of today, the Azure Purview Data Map is a 24×7 service – no option to pause the service. Which means – please check your Azure subscription type during the preview period!
It now includes the updated eligible Azure subscription types:
Azure Purview Data Map: The 4 free capacity units are only available for customers on the Pay-As-You-Go (MS-AZR-0003P), Microsoft Azure Enterprise (MS-AZR-0017P),Microsoft Azure Plan (MS-AZR-0017G), Azure in CSP (MS-AZR-0145P), and Enterprise Dev/Test (MS-AZR-0148P) offer types. Free quantities are applied at the enrollment level for enterprise customers. Free quantities are applied at the subscription level for pay-as-you-go customers.
Scanning and Classification: The 16 vCore-hours of free scanning are only available for customers on the Pay-As-You-Go (MS-AZR-0003P), Microsoft Azure Enterprise (MS-AZR-0017P), Microsoft Azure Plan (MS-AZR-0017G), Azure in CSP (MS-AZR-0145P), and Enterprise Dev/Test (MS-AZR-0148P) offer types. Free quantities are applied at the enrollment level for enterprise customers. Free quantities are applied at the subscription level for pay-as-you-go customers. Note: Azure Purview provisions a storage account and an Azure Event Hubs account as managed resources. This may incur separate charges that in most cases will not exceed 2% of charges for scanning. Refer to the Managed Resources section in the Azure portal within Azure Purview Resource JSON.
Pricing information for Azure Purview (screenshot taken 2021-02-01 9am CET)
So now – What is my subscription type? Where do I find this information?
You need the appropriate rights to view the subscription details in your Azure Portal. Open the Azure Portal (https://portal.azure.com) and open your list of Subscriptions and head over to the Subscription details.
In one of my Azure Sponsorship subscription, the Offer ID (subscription type) is MS-AZR-0036P = not eligible for the free Purview preview.
The Azure documentation page Understand Cost Management data (link) contains the list of available Azure offers and subscription types.
Updates to this blog post:
2021-02-27: Clarification of billing of Azure Purview
On Tuesday 23rd, February 2021 I got an email titled with Clarification of Billing for Azure Purview and free quantities in January and February 2021.
The email contains information about the fact that the free capacity units costs during the preview time were sometimes charged incorrectly. AND: Microsoft will refund these incorrectly billed charges back. What a nice surprise!
Refunding information
2021-02-27: Extension of the Azure Purview Free Preview Period
Another nice surprise went live on Saturday 27th February. The Purview team tweeted (link to the tweet) that the 4 Capacity Units of the Data Map will stay free until May 31, 2021! Thank you Purview Team!
To conclude
Get to know your Azure subscription type & check your Azure credits balance!
Another week and other data links for you! I hope you had a great weekend – I spent mine outdoors – long walks and long runs.. and it was great to be outside in the snow!
Data Lake Architecture – Bill Inmon published a rather controversy article about Data Lake architectures and how and if the work with Data Warehouses – DATA LAKE ARCHITECTURE | LinkedIn
Yesterday, the new Power Platform release wave was released. The 2021 release wave 1 plan (puh 🙂 ) includes the features planned for April 2021 through September 2021.
In personally prefer the web page since it is easier to browse through the different chapters and features, but there is also a PDF file available for download.
What is planned for Power BI?
The list of new features is long, therefore I will not repeat and copy it from the release plan itself. But I would like to highlight some of the features:
This list is only a small excerpt of the huge list of features planned. Have a look at the web page and re-visit it from time to time. There are updates happening from time to time!
During the writing of my latest blog post (What’s new in Azure Synapse Analytics?), I found a very interesting entry in the update feature list: Azure Purview Integration.
Azure Purview Integration in Azure Synapse is announced
And this really got my attention – use Azure Purview to find your data artefacts and use it for exploration… sounds nice.. let’s try it!
Configuration – Connect Azure Synapse to Azure Purview
In the Manage Hub of Azure Synapse, the new menu entry Azure Purview (Preview) is the entry point to connect your two Azure services.
Azure Purview integration in Azure Synapse
You need Contributor rights in Azure Synapse and access to your Purview instance. In the connection dialog, use the browse option or enter the Purview ID into the manual section.
Connect to a Purview account
When connected, the config section allows you to edit the mapping (i.e. change it to another purview instance), refresh (I am not sure what this action really does, but yes, refresh the connection) or disconnect the mapping. With the link on the purview instances name you’ll directed to the Purview studio.
Connection successful – My Purview account is connected to my Azure Synapse Workspace
Use the Purview integration in Azure Synapse
According to the documentation, you can use the Purview integration to discover data registered and scanned by Azure Purview …
This integration search is implemented the different hubs / development areas in Synapse Studio. It is available in the Data, Develop and Integrate hub in Synapse studio. After the Synapse / Purview connection is configured, a click into the search box allows you to select the search scope – let it be Purview or the (Synapse) Workspace.
Search in Azure Synapse with Purview connection enabled
Let’s search for the term Customer in our Purview account. The results are listed in the Purview search tab which is similar to the search results tab directly built-in Purview Studio.
Result list – Purview entities in Azure Synapse
For the demo of the functionality, I first open the Customer table (in an Azure SQL database). The detail page for a table in an Azure SQL database looks very familiar to the one in Purview directly. Purview features to browse your data assets are there as well. If your Purview credentials allow, you are also able to edit the properties of the entity (see my comments about security later in this blog post).
Start with an Azure SQL table
Detail page of an Azure SQL table in Azure Synapse (using Purview integration)
With two differences….There is a Connect and a Develop menu. Remember, we are located on the Azure SQL table detail page. With the Purview integration
Menu actions available for an Azure SQL table
As you can see in the screenshot, using the Connect menu allows you to create a linked service definition or a New integrationdataset. For the definition of a new integration dataset, you’ll need to create the linked service first. At the moment (2021-01-27) and according to the documentation, the Synapse/Purview integration is not able to infer if there is an existing linked service or integration dataset.
Options in the Connect menu in Azure Synapse/Purview integration
In the Develop section, with an Azure SQL table selected, we’ve got the option to create a Newdata flow.
Azure Purview integration in Synapse -> directly create a new data flow based on an Azure SQL table entity in Azure Purview.
Or with a file in a Data Lake
Switching back to the results list and next – open the wwi-dimcustomer.csv file in the ADLS Gen2 account. The overall detail page looks very similar to the one of the Azure SQL table, the Develop menu contains some more actions.
Details page for a file in the data lake (Azure Synapse / Purview integration)
For the selected entity (file in the data lake), the Develop menu allows you to directly start with your data exploration either using SQL OPENROWSET options, integrated Spark or a new data flow.
Develop options for a file in the data lake (Purview entity – Azure Synapse integration)
How about the credentials / security?
Based on the documentation page, the Purview integration in Synapse requires you to have Purview access. Synapse passes-through your Purview permissions.
To read more about Azure Purview roles (and their permissions in the catalog), I recommend you to have a look at the documentation.
To sum it up.. Better together? yes? no? YES!
In my opinion, the Purview integration in Azure Synapse is great step into the right direction.
With Synapse positioned as the central analytics platform, the direct access to Data Catalog information without leaving Synapse Studio is great.
Next, the immediate creation of linked services, integration data sets, or data exploration at your context menu is really a this makes it really easy to integrate data into Azure Synapse.
Stay curious, catalog your data artefacts AND enjoy your day! 😉
The pace of innovation in a cloud environment is a high one. Being updated almost continuously, it is somehow hard to follow all the announcements.
In Azure Synapse, there is one menu entry that informs us about new features, updates and lists the release notes. The update icon is available in the main menu and highlighted in the next screenshot.
The updates menu in Azure Synapse Analytics
Whenever you open the updates menu, you’ll get to a list of the latest updates.
The list of latest updates in Azure Synapse Analytics
Another week and other data links for you! I hope you had a great weekend – I spent mine outdoors – long walks and long runs.. and it was great to be outside in the snow!
I did not know that page – an overview about Connectors in Power Query – listed by source system PLUS the availability in the Power Query system (Excel, Power BI datasets/dataflows, PowerApps, Customer Insights and Analysis Services). This link is definitely in my bookmark-shortlist! – List of all Power Query connectors | Microsoft Docs
