CIO World spoke with Andrew Martin, Regional Leader, South Asia at Databricks
There will be an explosion of growth and innovation in the data and AI ecosystem in APAC more than ever before – volume of data, variety of data and the speed at which it needs to be used is growing at a very fast rate. In particular, the level of innovation in APAC and the demand for ways to use data is showing very exciting opportunities that will enable companies to help solve some of the world’s biggest problems.
We are seeing companies that have established ways to address income inequality, climate change, solving chronic diseases and digital disruption. With all of that demand and language, companies need to think about succeeding and outpacing the competition to man the ability to use data in a way that supports that growth, creates huge amounts of opportunities but also presents a lot of risks and considerations companies can look at too.
When it comes to privacy in particular, more broadly, companies are going to have to establish ways to think about their architecture – how their future state architecture will enable their data teams to use data. Then, they will have to build a strategy circulating around risk compliance, governance and security.
Data Privacy & Data Sharing Landscape of APAC In The Next 5 Years
Taking a look at industries, there is common ground in financial services more than ever before because of the security and governance requirements along with the demand for higher volumes of real-time data. It can be seen that industries like financial services are really paving the way for enabling data teams to innovate whilst making sure governance, security and privacy are built in.
With that said, there are a few trends that are emerging that are going to change the way companies think about data and therefore, how they are going to manage privacy as well. When it comes to the trends in laying this foundation, there are tens of thousands of companies out there and many of which are actively considering data use in APAC. When thinking about the ways in which data has been used, it has been observed that startups, enterprises and SMEs are adopting CLOUD; how they are migrating their data into the cloud and how to embrace open-source technologies to futureproof their architecture.
The next thing, when moving on to the cloud, is what companies want to do with it and today, one of the biggest challenges is that many companies have established some type of a data warehouse that will allow them to provide big solutions for their business. They then need to maintain that data and potentially move that data to the cloud so they can continue providing business with real-time insights and intelligence.
More than that, we are seeing a lot of high volume and unstructured data give rise to the ability to look at the reproduction machine learning, exploratory insights and streaming analytics. The question is then, how do companies continue to do all that without having different access points or separate platforms for data scientists, data engineers and data analysts in the company. This makes it extremely difficult for many companies to have some kind of data warehouse and database to establish governance and security that will ensure that as the business continues, it can be assured that data governance, security and privacy has all been considered to avoid risk and enable innovation.
With all of that being in place, the question will then be, what are actual problems the companies want to address. If we look across all industries, there is a range of used cases that CIOs, CEOs and CFOs are now thinking about. Each one of these used cases must be thinking about ensuring that data governance, privacy and security has been implemented. Doing this quickly and establishing consistency means that companies can innovate faster knowing that privacy is built in and if what continues right now is to happen in the future, companies will not run the risk of breaching privacy or putting their company at risk. This is because the volume and variety of these dataset cannot scale if they continue with the same solo architecture which they use today.
Influence Of The Introduction Of Databricks’ Lakehouse Onto GCP For APAC
Google Cloud Platform has focused on the ability to provide to customers that have requirements around multi-cloud and in particular, one other area which is really exciting is that Google has continued to embrace open-source and open standards like Databricks’ too. There is a rapidly growing demand to move cloud customers to go beyond all of the fantastic capabilities Google Cloud provides their customers and partners and simply look at ways data brings conjoined ways to collaborate with Google and their customers in order to accelerate innovation even further.
When companies start to work with Google Cloud customers, they will notice a strong demand for innovation without necessarily having to make large investments or go through complex buying processes. A lot of the data teams, just to be able to access the capabilities today, start using GCP tomorrow and then get results before going through some type of complex purchasing process. With Google Cloud, it is already fortunate enough to be able to see some fantastic customer customs, success stories around using Databricks.
Recently, Databricks announced some success stories thich they had with True Digital in Thailand where they were successful in using both Databricks and Google Cloud. Although Databricks just launched recently in Google Cloud, they have seen phenomenal demand and companies looking to embrace the multi-cloud, open-source and open standards.
Most Common Data Privacy Missteps In APAC
“The biggest opportunity right now is around data sharing. If you look at some of the large enterprise industries like telecommunications, for example, they are continuing to de-commoditize around APAC and looking for opportunities for new revenue streams,” commented Andrew Martin, Regional Leader, ASEAN & India, Databricks.
If companies intend to have new revenue streams, they have to take their existing data and look at new ways to monetise that and in particular, what other companies they can create partnerships with to compliment their offerings on revenue streams they seek to create.
One fascinating challenge, when coupled with the demands of first-party data and data transparency; how data is being managed appropriately, combined with the opportunities to innovate, companies can see banks, telecommunications, retailers and airlines having the potential opportunities to help each other serve consumers better. They can create better experiences for the consumers however, consumers did not necessarily give their permission for their data to be managed by companies that way. It is clear that technologies are now outpacing privacy requirements and compliance risks and it is creating continued demand in ways to enable data sharing whilst still maintaining data privacy of consumers and regulators. As companies continue to comply with data privacy, they can then collaborate data in the cloud using the same platforms to quickly get their products and services to help serve customers better.
Best Practices For The APAC Market
Generally speaking, whether it is a startup or a large enterprise, security and privacy teams need to be able to establish guidelines and in many cases, they already have. They need to be able to enable data teams to use data such that it complies with the necessary rules and regulations especially in data sharing. Making sure that there’s a way to manage that data in one place so that data is not solo and this is going to have a huge impact on the ability for the data team to use the data to partner with the compliance team in ensuring that their companies use the data correctly.
“The best practices for me personally would be to go back to the basics of what we are hearing customers tell us they need, which is, firstly, simplifying my architecture so that I have one platform to manage my data and secondly, help me to use data across multiple cloud platforms. Thirdly, embrace open standards and open source as companies find new ways to use data,” said Andrew Martin, Regional Leader, ASEAN & India, Databricks.
If companies simply use proprietary warehouses that potentially lock them in, they will be pushing away new cloud solution companies with data teams that want to innovate for a better data architecture for the company. If the platforms companies use are not embracing open source and open standards, that then creates risks to data privacy and compliance as they may use a separate architecture.
In essence, companies will need to have a good data architecture in place whilst also being open and embracing of open-source and open standards that will aid in the further innovation as well as the development of their architecture. With these best practices, companies will be able to service their consumers well without compromising on data privacy, security and governance regulations.
In today’s data-driven economy, a company’s ultimate worth is derived from the data it collects from its customers. This means that data is a valuable asset that should be safeguarded and preserved. Organisations frequently overlook the fact that the personal information of consumers gathered by them is merely borrowed. Consumers can use privacy laws to assert their rights, such as the right to be forgotten, and in some cases, they can reclaim ownership of their data. Businesses will have to establish constant compliance with data privacy, security, and governance regulations if they want to preserve their data as well as their consumers’ trust.