Big data and data science have been discussed for many years. However, there are lots of failed cases of implementation around the world. With the digitalization, it is no longer difficult to collect data but some organizations are now facing the new problem of drowning in a sea of data. In this article, I would like to share what I have been observed and experienced in the past decade in data analytics.
What’s your data purpose?
We are facing data overflowing with thousands of sources. However, there are many tools for gathering data but only a few of them doing analysis and discovering insight. Also, most of the vendors are promoting a bulky hardware list and data collecting tools. It is important to remember our focus should be creating value with your data and not just collecting the stuff for feeding storage boxes. The most painful business problem should be identified before doing data collection and analytics. Asking the key question is critical for driving the right direction and keeping the data analytic processes in the right track.
Make sense of your data
There are many analytics solutions provide extremely in-depth data but no additional value is being created. Therefore, there are data lakes, data warehouses and data silos as ocean in volume but no one is able to draw a report, a dashboard or an analysis easily. Another example is running a report for basic analysis for 3 hours a day.
What’s going wrong? You have all the data in the corporation but you can use it effectively.
For different roles, there are different handling methods for the data ocean. If you are a data scientist, you would enjoy in swimming and tasting the data. Nevertheless, it is important for a data scientist to identify whether the portion or a set of data is valuable for the organization for further processing and analysis. Also, data could be separated into smaller subsets for data analytics by the smaller and controllable scope. A data scientist may use data science tools or programming codes to do some basic statistics on the data and studying the data for the data profiling and data gap analysis prior to any other data analytic activities.
However, if you are a business user / manager, you would like to have relevant data to solve your business problem timely. Thus, they don’t need to have the full set of data in their organization like a data scientist. A business user should focus on the data that can help you achieve your business goals, projects, and targets. For “Citizen Data Scientist” (Power Business User) may take Business Intelligence tool to filter and manipulate your data to show only relevant information as the first step of making sense on your data.
Create value from your data
Nowadays, there are lots of great tool but many are just focused on data visualization. Charting is great for providing quick access of trends but it doesn’t tell much including the story behind. Do NOT stop analyzing your data in-depth after a beautiful chart being shown. It should be the starting point for further analysis. Moreover, there are more & more applications for automation based on insights. To take an example, I have been working on some predictive analysis on commodities and drive some robotic trading activities by our prediction results on price changes in the commodity market.
Data Analytic as Daily Assignment
There are some service companies in the world mainly in Asia, claiming that they are making the perfect solutions and fit for future after their once-off service delivery. The business world is always changing and your business is expected to grow. In 2020, there is the coronavirus crisis and it is NOT possible for the predictive model and any data analysis done in last year and still perfectly fit for today. Data analytic is always a continuous journey in parallel with your business operation.
Conclusion
In order to minimize the risk for the Data Science journey, it is better to follow a Data Science project flow being used by many different experts around the world. You can find another article of mine with the 5 steps BADIR approach:
https://samuelsum.com/2019/07/03/the-road-to-analytic-success-5-steps-badir-approach/
(Remark: The BADIR approach is originated by Piyanka Jain)