A featured contribution from Leadership Perspectives, a curated forum for enterprise technology leaders, nominated by our subscribers and vetted by the CIOApplications Editorial Board.

Citi

From Big to Small Data

Only three or four years ago, ‘big data’ was the buzzword of the moment. Today, the ‘big’ in big data is no longer important: we have quickly moved to an era in which insights discovery is at the heart of strategy, for businesses, governments or individuals. But what happened to ‘big data’, and what new dilemmas have emerged?

Big data has not gone away: the problem was that data has become too big. According to the International Data Corporation (IDC), the global ‘datasphere’ will grow by ten times between 2016 and 2025 to 163 zettabytes (trillion gigabytes) . Not only does the volume of data continue to grow exponentially, but so too does the speed of data and the number of ways that we produce and consume data, not least with the increase in smart devices and the internet of things (IoT). The more data that is produced, and to which individuals and businesses are exposed, the more difficult it becomes to make sense of it. Ultimately, however, the challenge is to decide which data is important, and what you can do with it to create new insights or opportunities i.e. picking out the small data from the big data.

From Big Things, Little Things Grow

However many people or technology you employ, it is almost impossible to find that (small data) needle in a haystack, and use that needle to create something of value, without harnessing technologies such as artificial intelligence (AI) and machine learning (ML). Using ML, you can create algorithms to sift through data and work out what you can learn, while AI offers an alternative to human analysis, but on a far more scalable basis. Essentially, AI extends beyond the collation and identification of the right data to the ability to recommend or take an action that fulfils a useful function.

With the internet of things, it is not only watches, smartphones and other devices that we see as ‘technology’ that are doing this, but our domestic appliances and cars too.

Big data is not dead, but there is a greater realization that bigger is not necessarily better

The shift of AI from communication devices alone to all manner of devices, gadgets and appliances marks a blurring of the lines between humans and technology. No longer is our community made up of our family, friends and co-workers, but it extends to include these people and their devices. For example, a diabetes sufferer using her smart watch to monitor her vital signs becomes part of a data sharing community. With the ability to bring together thousands of people and data points, AI could replace years or even decades of research into how individuals with different profiles respond to certain diets, medication or lifestyles.

Despite the benefits, if sensitive data, such as medical or financial information amongst many others, falls into the wrong hands, or is used in the wrong way, the potential to harm as well as help society and individuals can be enormous. Unfortunately, as the quantity and influence of data in society increases on one hand, so too does the incidence and severity of cybercrime.

Distributed trust technology?

New technologies, such as blockchain or distributed ledger technology (DLT), are likely to play a vital role in creating an environment of trust that is essential to a data-driven society.

Currently, information or assets, such as photos, files or documents, are copied every time they are transmitted to another party. Once transmitted, the originator does not know, and has no control over, how it is disseminated or altered. This creates an issue of trust. How do you know the asset that you’re working with is the original, unadulterated version? Scale this up to entire communities and the problem of trust and integrity becomes a serious concern: at the very least ‘fake news’. Consequently, a critical challenge is how to harness the data for the benefit of individuals, businesses and societies whilst keeping both the data and the people to which it relates safe and secure.

In contrast, data exchanged through blockchain-based solutions exists in one iteration, whether a picture, document or contract etc. As this asset passes across the network from one node or entity to another, a unique stamp means that every action or amendment is visible, therefore creating trust and protecting community participants.

Big data is not dead, but there is a greater realization that bigger is not necessarily better. Instead, a fundamental objective in today’s data-driven society is to find the right data, and use it to create meaningful insights and accurately informed decisions. New and emerging technologies such as DLT and AI play complementary roles in achieving this. By building trust into the exchange of data, the greater the validity of decisions and actions that are based on it for individual, commercial and societal benefit.

The articles from these contributors are based on their personal expertise and viewpoints, and do not necessarily reflect the opinions of their employers or affiliated organizations.