Blog

Problem

Developing the input pipeline in a machine learning project is usually time-consuming and uncomfortable, and it might take longer than building the model itself. When dealing with massive datasets including thousands/millions of files, the input data pipeline can either be a game changer or a bottleneck depending on the architecture.

When datasets are too large to fit in RAM, a python generator-based technique can represent a significant barrier in training complicated GPU compute-intensive models. GPUs that are idle while waiting for data significantly slow down model training.

This is when TensorFlow TFRecord comes in handy.

We will learn how to utilise TensorFlow’s Dataset modules tf.data and tf.record to create efficient pipelines for photos and text in this lesson.

TFRecord

To record, data is stored as a binary record in a succession of protocol buffers. TFRecords files are incredibly read-efficient and take up little hard drive space. TFRecord is a method for storing data samples in sequential order. Each sample contains a number of characteristics.

We specify our features as a dictionary within the ‘write record’ method. In this scenario, we have picture data, the location of the item on the image, and the colour of the object, and we also want to keep the image data’s form.

Certain datatypes can be specified for a feature in TFRecord. For our features, we use a byte list and a float list.

Now we enter our real data into the features: we begin with data in digits, which is quite handy. We flatten them (‘.ravel()’) and provide them to the appropriate feature constructor. You might be wondering why we store picture data as floats. This is a design choice (oops! — see on for the consequences of this design choice), so we already save the picture data with colour values in the 0=val=1 range, so we can feed this immediately to the training. You’ll see that there are a few of spots suited for data conversions – assuming you stored it as uINT8 here, you can later convert it in the feeding pipeline.

The last thing we need is a means to shut down the writer and ensure that everything is written to disc. We introduced a close writer function to do this (as an aside, you might modify this to operate in combination with the Python ‘with’ statement).

That’s all. We will return to this point later: we presently do not write validation data separately from training data. One may expect a simple’split dataset’ method that we can use afterwards, however there is none with Datasets. This is acceptable given that tf.data is designed to operate with billions of records, and splitting billions of information in a certain way is difficult.

To record eliminates the need to read each sample file from the disc for each epoch.

The effectiveness of utilising TFRecord comes at a cost, in the form of the need to write a lot of complicated code to produce and read the records files.

We provide a solution to this problem by using Datum to handle TFRecords files.

Datum

Datum is a TensorFlow-based framework for creating an efficient, quick input pipeline. Datum is intended to create/manage TFRecord databases nearly entirely without the use of complicated programming.

Datum is intended to read and publish TFRecord datasets while also constructing a fast input pipeline that can be used for single GPU or distributed training with only a few lines of code.

Datum constructs a fast input pipeline using tf.data and tf.record.

TFData

The Dataset API enables you to create an asynchronous, highly efficient data pipeline to prevent data exhaustion on your GPU. It reads data from disc (pictures or text), performs optimal transformations, generates batches, and sends them to the GPU. Previously, data pipelines forced the GPU to wait for the CPU to load the data, resulting in performance concerns.

Installation

Datum may be installed with the command pypi

pip instal datum.

TFRecord export

If you don’t use datum, writing/exporting data to tfrecord format might get quite complicated.

Datum facilitates the export of datasets to tfrecord format. Datum has a few preset issue kinds that allow you to generate a dataset with a few lines of code without having to go into the inner workings of tfrecord and serialisation.

Import TFRWriteConfigs to define datum configuration for writing/exporting data to tfrecord

from datum.configs import TFRWriteConfigs

Define the splits information in the configs, splits names are important for a datum to automatically identify the splits data.

write_configs = TFRWriteConfigs()
write_configs.splits = {
"train": {
"num_examples": <num of train examples in the dataset>
},
"val": {
"num_examples": <num of validation examples in the dataset>
},
}

To transform the datasets, import the export API and problem type.

Different datasets serve different functions. An image classification dataset with merely a class label, for example, cannot be utilised for picture segmentation or detection. To facilitate conversion, datum distinguishes between issue categories for classification, detection, and segmentation tasks.

from datum.export.export import export_to_tfrecord
from datum.problem.types import IMAGE_CLF

Suppose we want to build a tfrecord dataset for an image classification task, the type for that is IMAGE_CLF

Convert the dataset to tfrecord format

export_to_tfrecord(input_path, output_path, types.IMAGE_CLF, write_configs)

Datum will transform and store the result dataset. In the output directory, there are tfrecord files and dataset metadata files. The exported tfrecord files may be imported as tf.data with ease. Datum load API was used to load the dataset.

Load the data as tf.data.Dataset.

Import the loading API as tf.data to load the tfrecord dataset.

Import dataset from datum.reader

To load the dataset, simply give the output path from the previous export state.

dataset = load(<path to tfreord files folder>)
train_dataset = dataset.train_fn('train', shuffle=True)
val_dataset = dataset.val_fn('val', shuffle=False)

Examples/cases in the dataset can be augmented before feeding into the model. It’s easy to preprocess and post-process samples in the dataset using pre_batching_callbackand post_batching_callback .

pre_batching_callback : Using this callback example can be processed before batching.

post_batching_callback: Using this callback example can be processed after batching. Examples are processed as a batch.

Suppose we want to augment the dataset, which can be achieved using the following pre_batching_callback

def augment_image(example):
image = tf.image.resize(example["image"], IMG_SIZE)
image = tf.image.random_flip_left_right(image)
image = tf.image.random_flip_up_down(image)
example.update({"image": image})
return exampledataset_configs = dataset.dataset_configs
datset_configs.pre_batching_callback = lambda example: augment_image(example)
train_dataset = dataset.train_fn('train', shuffle=True)

Put Datum to the test.

The usage of datum for transfer learning an EfficientNet-B0 model for an image classification project is demonstrated in this notebook.

Play with the Transfer Learning with Datum notebook to speed up your input flow.

Derive maximum ROI for your business from AI

Request Demo

Today AI’s impact and potential value are undebatable. Many businesses have already evaluated and experimented with AI in various forms. To the extent that applications of AI are actively investigated in healthcare, recruitment, law, and government. However, the adoption rate varies across industries and companies, but AI is becoming increasingly prevalent in the business world. However, AI boost and bust cycles are seen over time, which makes potential adopters wary. On one side, there are greater success stories of AI implementation, especially within the technology and healthcare domain. There are other traditional businesses where AI adoption is much slower than expected.

Technology adoption is an ongoing process of operational use of technology, according to Neumann, Guirguis & Steiner. I prefer this definition because it clearly defines adoption as not being achieved at a single point in time. It is also emphasized for the operational success of AI, especially in the business context of AI adoption. Any technology adoption depends on not just technological advancement but also external factors such as social, economic, and environmental influences. Technological advancement, for instance, is the ease of use, efficiency, value drivers, etc.; external factors are social, economic, and environmental factors. All these factors are unequivocally applicable to AI today, evidently enforcing their effects on us today, creating an environment that demands either adoption or failure.

Given the potential impact and value of AI, enterprise AI practitioners and managers need to approach adoption in a thoughtful and strategic manner. This may involve carefully evaluating business needs, identifying appropriate use cases, and considering the ethical and social implications of AI adoption.

By focusing on operational success and understanding the broader contextual factors influencing AI adoption, businesses can navigate the potential challenges and maximize the benefits of this transformative technology.

There are several frameworks and tools that exist to support it. There are a few popular frameworks to which one can refer.

1. Gartner’s AI Maturity Model: This framework provides a structured approach for businesses to assess their current AI capabilities and chart a path for AI adoption at the organizational level.
2. Microsoft’s AI Business School: This framework is designed to help businesses understand AI concepts and provide guidance for AI adoption through a series of case studies, videos, and tutorials.
3. IBM’s AI Ladder: This framework helps businesses to understand the key stages of AI adoption, starting with defining the AI strategy and ending with scaling AI across the organization.
4. Deloitte’s AI in a Box: This framework provides a modular approach to AI adoption, allowing businesses to choose the components that best suit their needs.
5. McKinsey’s AI Enablement Framework: This framework helps businesses to develop a strategy for AI adoption, define use cases, build the necessary infrastructure, and scale AI across the organization.

The main challenge with AI adoption lies in its execution, as it has remained theoretical and difficult to implement in practical situations. The primary issue is the inability to simplify complex concepts to a level that can be effectively executed.

In most AI adoption models or frameworks, best practices for organizational readiness are suggested for three key areas: people, process, and technology. It is important to note that technology readiness also includes data readiness. These three areas are often referred to as the “golden triangle” of AI adoption.

The golden triangle of people, process, and technology is widely recognized as a useful framework for analyzing socio-technical systems. Each component of the triangle – such as the development of people’s skills, the importance of processes, and the fit of technology to business needs – is equally critical to the success of AI adoption. These components are intertwined and can impact each other in numerous ways. For instance, in the AI technology context, the technology selected by a project team may influence the nature of AI tasks and, consequently, the processes to which these tasks contribute, which in turn can impact the people involved, including project stakeholders. Understanding the interplay between these three components is essential for successful AI adoption.

The data is an extension of technology in this golden triangle. The importance of data, its availability, quality, and governance are pre-requisite for the concept of a data-driven culture in an organization. The successful adoption of AI requires data to be first understood, available, and managed. The importance of data can be better illustrated by separating Data and Technology in the golden triangle as below.

The people who drive processes can decide the relevance of data. Data also determines the output of AI technology and drives the direction of AI development in an organization. To reflect all this importance of data, let us extend the triangle to a square, including data at the apex.

 

Stages of AI Maturity

AI Adoption within an organization depends on mechanisms that vary based on the current level of AI maturity. There are three stages of AI project maturity such as AI Experimentation, AI Adoption, and AI Production.

• AI Experimentation:

An early stage of AI, where use cases need to be identified by business stakeholders, the success criteria need to be defined for each use case and overall. Then, a strategy to execute these use cases in terms of people, process, and technology needs to be evaluated.

• AI Adoption:

A transitioning stage from research to deployment. This is the hardest phase to deploy an AI model in an operational setting. There could be several operational challenges to overcome at this stage, such as ensuring operational data is readily available, quality controls are in place, etc. Many organizations also need to gain all the required skills to operationalize AI models.

• AI Production:

This is a production phase where proven AI models are integrated with production applications.

Let us understand how interaction of these PPDT model influence AI project maturity.

At the experimentation stage, it is required to match the technology to the kinds of data available to address the business problem that stands out (model testing and selection). During the adoption phase, the link to people is emphasized, as the data which captures expert judgment of the quality of AI outputs is needed to prove the technology’s value. Once the technology is mature, the link between data and processes must be built into the organization’s MIS infrastructure to maintain the supply of operational data.

Conclusion:

New age AI platforms have significantly reduced the challenges to AI adoption in an enterprise allowing them to be confident about adopting AI in an organization based on their AI maturity. Subex’s cutting edge platform (Such as HyperSense AI) enables enterprise customers to make faster, better decisions by leveraging AI across the data value chain. The platform allows users without coding knowledge to easily aggregate data from disparate sources, turn data into insights by building, interpreting, and tuning AI models, and effortlessly share their findings across the organization, all on a no-code platform.

8 Keys to AI Success in an Enterprise

Download E-book Now!

Abstract

Consistent asset health across various levels, from cell sites to regions, is crucial for telecom carriers to ensure ongoing operations. However, proactively detecting aberrant patterns such as equipment breakdown remains a significant problem. This article, based on Subex’s extensive implementation expertise, covers the key processes required to instal effective pattern detection solutions. It also looks at certain high-impact pattern identification use cases that have consistently generated value for telecom operators.

Introduction

High operational efficiency is a key distinction for telecom operators that want to build their brand name on consistency and service availability. Unusual consumption patterns in the call, Internet, and SMS services might reveal underlying concerns that could lead to greater ones. Consider how variations caused by power outages, technological problems, or competition growth might result in SLA breaches and revenue loss. To keep ahead of operational interruption, area managers must regularly monitor key performance indicators (KPIs) across many cell sites, clusters, and regions.

To detect abnormalities, most CSPs rely on manual and reactive methods. Any attempt to delve into the fundamental cause and take corrective action necessitates the usage of standard dashboards or the perusal of long performance analytics reports. These methods are prone to errors and frequently fail to account for shifting patterns, seasonality, and discrepancies in the data.

Automating pattern detection

Automated machine learning technologies can assist carriers in addressing the aforementioned difficulties. Data-centric frameworks are used to collect and cleanse historical data, which is then automatically fed into machine learning models that can learn from big datasets. This reveals tendencies and, as a result, forecasts daily consumption levels throughout each geographical area.

Consistent tracking is an important component of an automated system. Within a certain location, all deviations of real KPIs from predefined values are calculated. Significant deviations or abnormalities are immediately reported in a dashboard for prompt repair by the relevant area manager.

Implementation approach

For huge data, Apache Hadoop boasts industry-leading distributed processing capabilities. As a result, Hadoop is an effective tool for converting XDRs into trainable datasets that can be fed into powerful machine-learning algorithms to detect abnormalities. These anomalies in telecom pertain to spikes and decreases in everyday usage. The following are the major steps of a machine learning-driven pattern identification solution:

Step 1:Ingest the data — An automated process is configured to transmit data from numerous sources to Apache Hive (data warehouse) via real-time streaming systems that collect information such as session length, charges received, location, and so on. To build analytical datasets, this granular data must be analysed, wrangled, and organised.

Step 2:Select an algorithm — When developing the framework, it is critical to select the appropriate algorithm, one that scales well and solves data complexity. When comparing several forecasting approaches, some essential criteria to consider are:

• • Capability to identify outliers and missing data
  • Capability to learn at speed and scale as needed
  • Handling significant changes in time series
  • Integration and automation capabilities
  • Customizability and interpretability

Following a thorough examination of regression models such as autoregressive integrated moving averages (ARIMA), Holt-Winters, and others, it was discovered that Facebook’s Prophet algorithm meets the aforementioned requirements and can be implemented fast. It enables users to quickly personalise predictions. The model may also be supplied with domain information via human-interpretable parameters, enhancing forecast accuracy even more. Subex has exhaustively analysed the performance of Prophet after working with a massive quantity of real-world telecom data and discovered that it is 8-10% more accurate than traditional methodologies.

Step 3:Deploy the model – Once the Prophet algorithm has been developed in R, the input data streams must be set up using Hadoop to R connectors. To keep up with the newest developments and trends, the model should be updated daily with consumption statistics, data from each cell site, and so on.

The cell sites can be organised into clusters or regions and examined accordingly to establish the hyperparameters. Alternatively, hyperparameters for groups based on cell site category (2G, 3G, or 4G), K-means clustering, or other classification approaches can be defined. The data is then divided into training and test sets in a 4:1 ratio.

For example, the first 80 days are devoted to training sets, followed by 20 days of assessment. Once the hyperparameters for each group have been determined based on the trend analysis of the group, the Prophet model may be repeated for each cell site using training data. Any break of a threshold, whether a spike or a decrease, might be marked as an abnormality. The abnormal cell location will be marked, and the information will be put back into the Hive for viewing on the dashboard.

Step 4:Visualize the results — The Apache Hive table (created by R’s machine learning package) contains data on all cell locations that saw spikes or drops in a single day.

This data comprises the preset criteria, predicted values, and actual use metrics, as well as the amount of the difference between forecasted and actual metrics. The spatial hierarchy associated with anomalous places is also noted. The Hive tables are combined with a dashboard platform like Qlik Sense to allow for speedier decision-making and simpler visualisation.

Benefits of pattern detection

Pattern recognition based on machine learning assists telecom operators in transforming the time-consuming, manual, and reactive monitoring of multi-level operational assets into an end-to-end, touchless, and highly efficient process.

Subex assisted a large African communication service provider in implementing pattern detection to improve on-site assets and use monitoring. Among the primary benefits obtained were:

• Effort savings — Effective anomaly detection saves 20-30 man-hours each month by taking seasonality, trend, holidays, and change points into account.
• Increased productivity – By just monitoring data on the dashboard, area managers may determine the core causes of irregularities across cell sites.
• Site enhancements – As a result of the multi-country deployment, the telecom is now able to log abnormalities as they occur. They discovered roughly 30 unusual instances spread throughout 1600 locations, over 100 clusters, and 8 regions.
• Useful insights — They could correctly identify the causes of usage drop like power interruptions, airtime recharge issues, client relocation, competitor inflow, extreme pricing spikes, and so on.
• Quicker decision-making – The telecom may undertake site-specific remedial measures such as refill schedule revisions, tailored marketing campaigns, and site renovations in real-time.

Case studies for Subex: Pattern detection applications in telecommunications

Reduced consumption data prompt immediate response, increasing customer retention by 90%.

A telecom operator discovered various irregularities in their international call use across significant cell sites using Subex’s Analytics Center of Trust. A closer investigation found that many of its dual sim users had switched to a competitor’s profitable overseas package. In response, the telecom quickly launched an appealing counter-bundle, allowing it to retain 90% of at-risk subscribers. The telecom was able to save USD 250,000 in monthly losses as a result of this.

Anomaly detection assists telcos in detecting revenue leakage and underlying fraud.

A major telco’s critical 4G cell site was identified for unusually high usage of the airtime credit service (ACS). Root cause analysis revealed that 5 individuals were abusing one of the ACS channels, borrowing USD 10,000 in credit many times. Within 24 hours, the Subex system detected the scam and took prompt action by blocking the fraudsters’ accounts. Following that, the security weaknesses in the ACS channel were fixed, saving USD 120,000 in damages.

Using market forces to increase client loyalty

When a telecom operator saw a dramatic increase in data service demand in a certain area, he proceeded to investigate regional abnormalities. According to reports, many new consumers had joined as a result of a service outage in a competitor’s network. Customers’ proclivity to carry numerous SIM cards resulted in roughly 10,000 additional members. The operator quickly launched a campaign to boost consumer loyalty to its data services, resulting in considerably increased data income.

Minor network availability issues result in significant cost reductions.

The availability of network cell sites is a key measure of network health. However, many telcos are unaware of how low network availability affects their company. Subex built a pattern identification system that defined KPI levels and established automated monitoring processes for a telecom operator experiencing network availability concerns. Daily anomaly detection of even small network failures provided the operator with the necessary knowledge to act quickly, allowing them to save USD 1 million each month.

 Using pattern recognition to improve the customer experience

Faced with several customer complaints about poor network performance during late hours, a CSP decided to employ pattern recognition to determine the actual reason. According to the model’s findings, consumer complaints were 2.3 times greater than in other sectors. However, because use trends revealed only a few subscribers lived at the cell site, they could rule out network congestion as a potential cause.

Conclusion

Maintaining asset health through continuous monitoring is a critical competence for telecoms seeking to maintain a competitive advantage through robust service delivery. Pattern identification solutions based on automated and machine learning are developing as a beneficial technique to monitor use trends while utilising powerful analytics and sensible visualisation. CSPs should establish the correct business cases and schedule solution deployment to achieve strong returns on investment. Subex has extensive implementation expertise as well as industry-leading solutions to assist operators in automating pattern identification for increased income and efficiency.

Derive maximum ROI for your business from AI

Request Demo

Conversational Artificial Intelligence (or Conversational AI) is a set of technologies underpinning automated messaging and speech-enabled systems that enable human-like interactions between computers and humans.

Conversational AI refers to any computer that can be spoken to and is most commonly encountered today via chatbots and voice assistants.

 

What is the definition of Conversational AI?

Conversational AI (artificial intelligence) refers to systems that can “speak” to people, such as virtual assistants or chatbots (e.g., answer questions).

Conversational AI apps are frequently utilised in customer support. They are available on websites, online retailers, and social media platforms. AI technology may significantly improve the speed and efficiency with which consumer questions are answered and routed.

 

How does conversational AI work?

Conversational AI is essentially powered by two functionalities. The first of these is machine learning. Simply said, machine learning means that the technology “learns” and improves as it is utilised. It gathers data from its exchanges. It then utilises that knowledge to develop itself over time.

As a consequence, your system will perform better six months after you add it to your website, and even better a year later.

The second is known as natural language processingv or NLP. This is the method through which artificial intelligence comprehends language. It can progress to natural language generation after learning to identify words and phrases. This is how it communicates with your consumers.

For instance, if a consumer approaches you on social media inquiring when an item will be sent, the conversational AI chatbot will know how to answer. This is because it has experience addressing similar queries and recognises which words perform best in response to shipping questions.

Although the theory may appear difficult, conversational AI chatbots provide a highly easy client experience.

 

What are the Components of Conversational AI

Conversational AI refers to any machine that may be spoken to. A chatbot on a website or social messaging app, a voice assistant or speech-enabled device, or any other interactive messaging interface might be used. Through discussion, people can ask questions, acquire views or suggestions, complete transactions, get help, or achieve other context-dependent goals.

 

Conversational AI brings together five technology components

1. Automatic Speech Recognition (ASR)
2. Natural Language Understanding (NLU)
3. Dialogue Management
4. Natural Language Generation (NLG)
5. Text to Speech (TTS)

 

Automatic Speech Recognition (ASR) :

Speech Recognition is the computer-based processing and recognition of human voice (Automatic Speech Recognition). It is the process of translating a voice signal to a series of words using computer software and an algorithm. It converts voice to text in conversational AI.

 

Natural Language Understanding (NLU) :

It is a subset of Natural Language Processing (NLP) that entails converting human language into a machine-readable format. NLU is concerned with a machine’s capacity to comprehend human language. NLU is the process of rearranging unstructured data so that machines can “understand” and evaluate it.

 

Dialogue Management (DM):

The key component of Conversational AI is Dialogue Management, which receives input from the ASR and NLU systems, interacts with external knowledge sources, and generates messages to be sent to the user. The dialogue management method consists of two major tasks:

Modelling dialogue: Keeping track of where the conversation is at.

Making decisions regarding the next system action using dialogue control.

In general, it directs the flow of communication between the agent and the user.

 

Natural Language Generation (NLG):

It is a subset of the Natural Language Process (NLP), which is defined as the “process of creating meaningful phrases and sentences in natural language form.” It develops narratives that describe, summarise, or explain structured data supplied in a human-like manner.

 

Text-to-Speech (TTS) :

Conversational AI has reached its conclusion. The text answer generated by the NLU and NLG stages is converted to natural-sounding speech using a text-to-speech (TTS) system. It operates in the inverse of the ASR system. Below diagram depicts TTS architecture :

Conversational AI may be the future of numerous day-to-day living activities as technologies and processing capacity advance. Because of the speedy responses it gives, conversational AI will improve consumer happiness.

 

Conversational AI use cases

When people think of conversational artificial intelligence, they often think of online chatbots and voice assistants for their customer support services and omnichannel deployment. Most conversational AI apps include comprehensive analytics in the backend software, which aids in providing human-like conversational interactions.

Experts believe existing conversational AI applications to be poor AI since they are focused on executing a relatively restricted set of activities. Strong AI, which is still a theoretical idea, focuses on a human-like awareness that can tackle a wide range of activities and issues.

Despite its restricted emphasis, conversation AI is a very valuable technology for organisations, assisting them in becoming more profitable. While an AI chatbot is the most common kind of conversational AI, there are several more applications throughout the company. Here are a few examples:

• Online customer support: Throughout the customer journey, chatbots are replacing human representatives. They respond to commonly asked questions (FAQs) regarding issues such as shipping or give individualised advice, such as cross-selling items or recommending sizes for users, altering the way we think about client involvement across websites and social media platforms. Messaging bots on e-commerce sites with virtual agents, messaging applications like Slack and Facebook Messenger, and jobs often performed by virtual assistants and voice assistants are examples.
• Companies may become more accessible by lowering entrance barriers, especially for people who use assistive technology. Text-to-speech dictation and language translation are common Conversations AI functions for these groups.
• HR procedures: Conversational AI may be used to optimise several human resources operations, such as employee training, onboarding processes, and updating employee information.
• Health care: Conversational AI has the potential to make health care services more accessible and cheap for patients while also enhancing operational efficiency and streamlining administrative processes such as claim processing.
• Internet of things (IoT) devices: Nearly every household now has at least one IoT gadget, ranging from Alexa speakers to smartwatches to cell phones. To engage with end users, these gadgets employ automatic voice recognition. Amazon Alexa, Apple Siri, and Google Home are all popular apps.
• Computer software: Conversational AI simplifies many office chores, such as search autocomplete and spell check when you search anything on Google.

While most AI chatbots and applications still have minimal problem-solving abilities, they can save time and money on recurring customer support engagements, freeing up staff resources for more engaged client interactions. Overall, conversational AI apps have been successful in simulating human conversational interactions, resulting in increased levels of consumer happiness.

Get started with HyperSense AI!

Request Demo