Turning the Ignite 2021 shock into a growth opportunity

Microsoft’s Dynamics 365 is a powerful business ecosystem of business apps such as CRM for sales and customer service, Business Intelligence, no-code apps, many integrations, and much more features, all based on a robust database layer, common for almost all services, called Dataverse ( previously known as Common Data Service). 

Accessing to raw data

This database layer is nested in the capabilities of Microsoft SQL Server, a functionality that has been, until recently that is, non-transparent to the users. That meant that anyone required to get access to raw data, for example, to train machine learning models, was presented with very limited options, such as API access, Open Data Protocol (OData) access, or Data Export Service (DES). 

The latter was the best option when it came to exporting a large amount of data, as well as a solid solution on which many integration scenarios were based, since it replicates all data to a dedicated database in Azure SQL without any additional cost, taking aside the Azure cost of the target database. 

There was another solution made available as of February 2020, enabling the export of Dataverse data to Azure Data Lake that, most admittingly, didn’t seem to bring any additional value to the already given DES at the time.

Adapting to change

That was until about a year ago, during Ignite 2021 when Microsoft announced it would be deprecating the DES after 12 months in favor of Azure Data Lake’s successor, the Azure Synapse Link for Dataverse, the very service that we kept overlooking all this time! 

After the initial shock, we had a meeting with our team and started designing the transition to the new modus operandi, having in our mind that at least we have a good 12 months to prepare. Alas, that was not the case! 

Notwithstanding that the announcement was stating a transition period of 12 months, the DES was removed from the Microsoft AppSource immediately, under the provision that it would continue to work for the existing customers, but it could not be used for new customers, leaving us no choice but to adopt the new solution immediately!

The impact on our business

As it turned out the adoption of Azure Synapse for our BI and ML solutions made us and our customers much stronger. Azure Synapse Analytics has proven to be an awesome product that combines Spark clusters (like Databricks), Serverless SQL, Dedicated SQL pools (similar to Azure SQL), and pipelines (similar to Azure Data Factory) in one place, empowering a whole team of data engineers, BI engineers, data analysts, and ML engineers, work on the same platform with minimal data movement. 

Moreover, we all feel now confident that Dataverse is put at the forefront of Microsoft’s data strategy, rather than a relic that had to be reluctantly supported as was the case with DES, with Azure Synapse having become the central data repository by extending the “links” from Azure Synapse Link for Dataverse to Azure Synapse Link for Azure Cosmos DB and Azure Synapse Link for SQL Server.

The new now; taking data-driven decisions seamlessly

Was the transition easy? Not at all, as we faced many challenges, including cost management (serverless infrastructure is billed based on the amount of data processed), the integration with Power BI, the refresh of the data, and the peculiarities of Synapse Link. However, we pulled through and have now put ourselves in a position of offering a much better data management strategy to our customers, while helping them make insightful data-driven decisions; having to deal with endless tables of hundreds of millions of records is not a concern anymore!

Azure Synapse Analytics now watches our back!