An ambitious partnership between academia and industry has delivered a breakthrough in software technology that will improve the quality, reliability and usability of numerical data from a variety of sensorial sources in a wide range of industrial application domains. The concept has been tested, verified and validated using a very large O&G industrial production and monitoring dataset.
The project is spearheaded by The Data Lab – Scotland’s innovation centre for data and AI, the Oil & Gas Innovation Centre (OGIC) and the software research and development company HyperDAP Ltd. The University of Aberdeen has also supported the software research and development process.
The result is the IDQI (Intelligent Data Quality Improver), an Ai-based algorithmic technology that can interpret and manage vast volumes of numerical data from heterogeneous sources.
The challenge
Industries are facing challenges in the handling, processing and interpreting of increasing quantities of data; many data-intensive industrial sectors are severely affected by this “data deluge” challenge and are therefore at risk of failing to extract valuable information from their production and other experimentally gathered datasets.
Data has become one of the most valuable assets a business can own, but without proper examination and interpretation it has very little value.
HyperDap has invested in its Intelligent Data Quality Improver technology to address the issue of numerical data quality, both in the oil and gas industry, and in many other industrial sectors.
The IDQI solution
The venture began in 2019 and was conducted at the University of Aberdeen for over a year, with the project demonstrating that numerical data can be assessed, measured, and improved in a wide framework of practical conditions.
This project has delivered the Intelligent Data Quality Improver, a highly innovative Artificial Intelligence algorithmic technology for data quality management and analysis. The IDQI, initially implemented as an algorithm technology for oil and gas data analysis systems, uses AI to guide the context-dependent quality improvement of big numerical production datasets from a variety of sensorial sources.
The HyperDAP team is a unique combination of experienced and young professionals sharing substantial expertise in AI and full stack development – ideally equipped to take this project and its underpinning software technology to the next level.
The innovation and uniqueness of the IDQI is threefold:
- It will measure numerical quality, proposing strategies to improve it if this falls below a given user-defined, context-dependent level.
- It will evaluate the effect of data quality improvement techniques, proposing alternative ones if dataset quality after improvement still falls short of its target.
- It will learn about user decisions in relation to proposed data quality improvement strategies, using this knowledge to guide and improve subsequent strategies.
These innovations will give IDQI algorithm users the possibility of optimising the quality of existing production datasets in context and of assessing the impact of different computational workflows using their results, enabling a much wider effective reuse of data than currently is the case.
The results
IDQI uses software technologies based on Artificial Intelligence in the management of massive datasets to overcome the limitations of current systems. An example of this is the lack of automated data quality evaluation and improvement functionalities and the automated suggestion of optimal workflows based on actual data quality.
The platform will address data quality and its improvement during analysis, focusing on the intelligent specification of different workflows to get meaningful results. It will automatically learn user decisions on machine-proposed workflows, using this knowledge to guide the optimal specification of subsequent workflows.
This innovation uses advanced AI techniques to perform the following:
- Improve production data quality and data analysis.
- Give users the possibility of running a number of different workflows in real time.
- Exploit machine suggestions to identify optimal solutions.
It will allow users to directly specify, program, and run their own workflows without depending on third parties and to exploit machine suggestions at their best. Current systems do not currently include automated mechanisms to evaluate and improve data quality, to recommend alternative workflows depending on evaluation results, or to learn user decisions and use them in the subsequent machine-generated proposal of further optimal actions in this context.
There is indirect positive environmental impact in the oil and gas sector that has been used as a testbed. The technology could help various leading-edge applications such as those analysing oil well integrity and performance to improve safety and decrease the likelihood of malfunctions. Operators will also be able to identify anomalies that generate environmental pollution from the subsurface into sea water.
Many sectors currently produce large numerical datasets daily, ranging from gigabytes to terabytes. The oil and gas sector is just one such producer, but it is far from being the only one. The IDQI project has the potential to revolutionise data interpretation across all sectors, bringing greater insight than ever before.
What they say
The project was subject to an independent Peer review as part of the Oil and Gas Innovation Centre approval process, where Industry experts are asked to comment on the innovativeness and applicability of the project. One reviewer wrote the following:
“Hydrocarbon accounting is an area of business in which small changes in effectiveness of data handling and systems quality can have a disproportionately large impact on financial gain / risk reduction / accounting robustness.
“As such, it is an ideal area for investment and the application of new technology in pursuit of those business gains. This area of business is also characterised by a large dominance of bespoke systems of very different vintages and technologies, and lack of competition in the service sector providing the niche but essential specialist services in this field. Thus, bringing in new ideas, new technology and new players in the service sector will be of benefit. Bringing in the latest computer science technology from academia has the potential to disrupt this market which is likely to be in need of some innovation. Good luck with the project. I look forward to hearing of its successes.”
In The Data Lab’s words
Gillian Docherty, The Data Lab’s CEO, said: “This wonderful project is a great example of how data, without processing and interpretation, can go to waste. This doesn’t just apply to the oil and gas sector and I would encourage industry leaders from any sector to get in touch with The Data Lab to discover how data can enhance your research and development work.
“Congratulations to everyone at HyperDap and the University of Aberdeen for their tenacity in seeing this project come to fruition. I look forward to seeing how IDQI is developed further.”
The post IDQI – Integrating Artificial Intelligence in numerical data quality improvement appeared first on .
