Trusted, high-quality data is essential to make informed decisions, deliver exceptional customer experiences, and stay competitive. However, maintaining quality is not quite so simple, especially as data volume grows. Data arrives from diverse sources, is processed through multiple systems, and serves a wide range of stakeholders, increasing the risk of errors and inconsistencies. Poor data quality can lead to significant challenges, including:

• Operational Inefficiencies: Incorrect or incomplete data can disrupt workflows and increase costs.
• Lost Revenue Opportunities: Decisions based on inaccurate data can result in missed business opportunities.
• Compliance Risks: Regulatory requirements demand accurate and reliable data; failure to comply can result in penalties.
• Eroded Trust: Poor data quality undermines confidence in data-driven insights, impacting decision-making and stakeholder trust.

Manual approaches to data quality are no longer sustainable in modern data environments. Organizations need a solution that operates at scale without compromising performance, integrates seamlessly into existing workflows and platforms, and provides actionable insights for continuous improvement.

This is where Tiger Analytics’ Snowflake Native Data Quality Framework comes into play, leveraging Snowflake’s unique capabilities to address these challenges effectively.

Tiger Analytics’ Snowflake Native Data Quality Framework – An Automated and Scalable Solution

At Tiger Analytics, we created a custom solution leveraging Snowpark, Great Expectations (GE), Snowflake Data Metric Functions, and Streamlit to redefine data quality processes. By designing this framework as Snowflake-native, we capitalize on the platform’s capabilities for seamless integration, scalability, and performance.

Snowflake’s native features offer significant advantages when building a Data Quality (DQ) framework, addressing the evolving needs of data management and governance. These built-in tools streamline processes, ensuring efficient monitoring, validation, and enhancement of data quality throughout the entire data lifecycle:

• Efficient Processing with Snowpark:
  Snowpark lets users run complex validations and transformations directly within Snowflake. Its ability to execute Python, Java, or Scala workloads ensures that data remains in place, eliminating unnecessary movement and boosting performance.
• Flexible and Predefined DQ Checks:
  The inclusion of Great Expectations and Snowflake Data Metric Functions enables a hybrid approach, combining open-source flexibility with Snowflake-native precision. This ensures that our framework can cater to both standard and custom business requirements.
• Streamlined Front-End with Streamlit:
  Streamlit provides an interactive interface for configuring rules, schedules, and monitoring results, making it accessible to users of all skill levels.
• Cost and Latency Benefits:
  By eliminating the need for external tools, containers, or additional compute resources, our framework minimizes latency and reduces costs. Every process is optimized to leverage Snowflake’s compute clusters for maximum efficiency.
• Integration and Automation:
  Snowflake’s task scheduling, streams, and pipelines ensure seamless integration into existing workflows. This makes monitoring and rule execution effortless and highly automated.

Tiger Analytics’ Snowflake Native Data Quality Framework leverages Snowflake’s ecosystem to provide a scalable and reliable data quality solution that can adapt to the changing needs of modern businesses.

Breaking Down the Tiger Analytics’ Snowflake Native Data Quality Framework

1. Streamlit App: A Unified Interface for Data Quality

   Serves as a centralized front-end, integrating multiple components of the data quality framework. It allows users to configure rules and provides access to the profiler, recommendation engine, scheduling, and monitoring functionalities – all within one cohesive interface.

   This unified approach simplifies the management and execution of data quality processes, ensuring seamless operation and improved user experience

2. Data Profiler

   Data profiler automatically inspects and analyzes datasets to identify anomalies, missing values, duplicates, and other data quality issues directly within Snowflake. It helps generate insights into the structure and health of the data, without requiring external tools.

   It also provides metrics on data distribution, uniqueness, and other characteristics to help identify potential data quality problems

3. DQ Rules Recommendation Engine

   The DQ Rules Recommendation Engine analyzes data patterns and profiles to suggest potential data quality rules based on profiling results, metadata, or historical data behavior. These recommendations can be automatically generated and adjusted for more accurate rule creation.

4. DQ Engine

   The DQ Engine is the core of Tiger Analytics’ Snowflake Native Data Quality Framework. Built using Snowpark, Great Expectations, and Snowflake Data Metric Functions, it ensures efficient and scalable data quality checks directly within the Snowflake ecosystem. Key functionalities include:

   • Automated Expectation Suites:
     The engine automatically generates Great Expectations expectation suites based on the configured rules, minimizing manual effort in setting up data quality checks.
   • Snowpark Compute Execution:
     These expectation suites are executed using Snowpark’s compute capabilities, ensuring performance and scalability for even the largest datasets.
   • Results Storage and Accessibility:
     All validation results are stored in Snowflake tables, making them readily available for monitoring, dashboards, and further processing.
   • On-Demand Metric Execution:
     In addition to GE rules, the engine can execute Snowflake Data Metric Functions on demand, providing flexibility for ad hoc or predefined data quality assessments. This combination of automation, scalability, and seamless integration ensures that the DQ Engine is adaptable to diverse data quality needs.
5. Scheduling Engine

   The Scheduling Engine automates the execution of DQ rules at specified intervals, such as on-demand, daily, or in sync with other data pipelines. By leveraging Snowflake tasks & streams, it ensures real-time or scheduled rule execution within the Snowflake ecosystem, enabling continuous data quality monitoring.

6. Alerts and Notifications

   The framework integrates with Slack and Outlook to send real-time alerts and notifications about DQ issues. When a threshold is breached or an issue is detected, stakeholders are notified immediately, enabling swift resolution.

7. NLP-Based DQ Insights

   Leveraging Snowflake Cortex, the NLP-powered app enables users to query DQ results using natural language, providing non-technical users with straightforward access to valuable data quality insights. Users can ask questions such as below and receive clear, actionable insights directly from the data.

   • What are the current data quality issues?
   • Which rules are failing the most?
   • How has data quality improved over time?
8. DQ Dashboards

   These dashboards offer a comprehensive view of DQ metrics, trends, and rule performance. Users can track data quality across datasets and monitor improvements over time. It also provides interactive visualizations to track data health. Drill-down capabilities provide in-depth insight into specific issues, allowing for more detailed analysis and understanding.

9. Data Pipeline Integration

   The framework can be integrated with existing data pipelines, ensuring that DQ checks are part of the ETL/ELT process. These checks are automatically triggered as part of the data pipeline workflow, verifying data quality before downstream usage.

How the Framework Adds Value

As organizations rely more on data to guide strategies, ensuring the accuracy, consistency, and integrity of that data becomes a top priority. Tiger Analytics’ Snowflake Native Data Quality Framework addresses this need by providing a comprehensive, end-to-end solution that integrates seamlessly into your existing Snowflake environment. With customizable features and actionable insights, it empowers teams to act quickly and efficiently. Here are the key benefits explained:

• End-to-End Solution: Everything from profiling to monitoring is integrated in one place.
• Customizable: Flexibly configure rules, thresholds, and schedules to meet your specific business requirements.
• Real-Time DQ Enforcement: Maintain data quality throughout the entire data lifecycle with real-time checks.
• Seamless Integration: Fully native to Snowflake, integrates easily with existing data pipelines and workflows.
• Actionable Insights: Provide clear, actionable insights to help users take corrective actions quickly.
• Scalability: Leverages Snowflake’s compute power, allowing for easy scaling as data volume grows.
• Minimal Latency: Ensures efficient processing and reduced delays by executing DQ checks directly within Snowflake.
• User-Friendly: Intuitive interface for both technical and non-technical users, enabling broad organizational adoption.
• Proactive Monitoring: Identify data quality issues before they affect downstream processes.
• Cost-Efficiency: Reduces the need for external tools, minimizing costs and eliminating data movement overhead.

Next Steps

While the framework offers a wide range of features to address data quality needs, we are continuously looking for opportunities to enhance its functionality. We at Tiger Analytics are exploring additional improvements that will further streamline processes, and increase flexibility. Some of the enhancements we are currently working on include:

• AI-Driven Recommendations: Use machine learning to improve and refine DQ rule suggestions.
• Anomaly Detection: Leverage AI to detect unusual patterns and data quality issues that may not be captured by traditional rules.
• Advanced Visualizations: Enhance dashboards with predictive analytics and deeper trend insights.
• Expanded Integration: Explore broader support for hybrid cloud and multi-database environments.

A streamlined data quality framework redefines how organizations ensure and monitor data quality. By leveraging Snowflake’s capabilities and tools like SnowPark, our Snowflake Native Data Quality Framework simplifies complex processes and delivers measurable value.

The post Building Trusted Data: A Comprehensive Guide to Tiger Analytics’ Snowflake Native Data Quality Framework appeared first on Tiger Analytics.

Three Roadblocks on the Journey from Data-to-Insights

In our work with several Fortune 500 clients across domains, we’ve observed the path to actionable insights extracted from data is hindered by a trifecta of formidable bottlenecks that often prolong time to value for businesses.

• The pressing need for personalized insights tailored to each user’s role
• The escalating complexities of dashboard development, and
• The constant stream of updates and modifications required to keep pace with evolving business needs

As companies navigate this challenging landscape, the integration of Generative AI (GenAI) into BI processes presents a promising solution, empowering businesses to unlock the true potential of their data and stay ahead in an increasingly competitive market.

Challenge 1: Lack of persona-based insights

Every user persona within an organization has different insight requirements based on their roles and responsibilities. Let’s look at real-world examples of such personas for a CPG firm:

• CEOs seek insights into operational efficiency and revenue, focusing on potential risks and losses
• Supply Chain Managers prioritize information about missed Service Level Agreements (SLAs) or high-priority orders that might face delays
• Plant Managers are interested in understanding unplanned downtime and its impact on production

Hence, the ability to slice and dice data for ad-hoc queries is crucial for gaining technical know-how. However, the challenge lies in catering to these diverse needs while ensuring each user gets relevant insights tailored to their roles. Manual data analysis and reporting may not pass the litmus test, as it can be too time-consuming and may not be able to provide granularity as desired by the key stakeholders.

Challenge 2: Growing complexities of dashboard development

Creating multiple dashboards to meet the diverse needs of users requires a lot of time and effort. It typically involves extensive stakeholder discussions to understand their requirements, leading to extended development cycles. The process becomes more intricate as organizations strive to strike the right balance between customization and scalability. With each additional dashboard, the complexity grows, potentially leading to data silos and inconsistencies. Dependency on analysts for ad-hoc analysis also causes more delays in generating actionable insights. The backlog of ad-hoc requests can overwhelm the BI team, diverting their focus from strategic analytics.

Managing various dashboard versions, data sources, and user access permissions adds another layer of complexity, making it difficult to ensure consistency and accuracy.

Challenge 3: Too many updates and modifications

The relentless need to update and modify the dashboard landscape puts immense pressure on the BI teams, stretching their resources and capabilities. Rapidly shifting priorities and data demands can lead to a struggle to align with the latest strategic objectives. Also, constant disruptions to existing dashboards can create user reluctance and hinder the adoption of data-driven decision-making across the organization.

Plus, as businesses grow and evolve, their data requirements change. It leads to constant updates/modifications– triggering delays in delivering insights, especially when relying on traditional development approaches. As a result, the BI team is often overwhelmed with frequent requests.

Empowering BI through GenAI

What if anyone within the organization could effortlessly derive ad-hoc insights through simple natural language queries, eliminating the need for running complex queries or dependence on IT for assistance? This is where the integration of GenAI and NLP proves invaluable, streamlining information access for all key users with unparalleled ease and speed.

At Tiger Analytics we developed Insights Pro, a proprietary GenAI platform to overcome these challenges and deliver faster and more efficient data-to-insights conversions.

In a nutshell, by generating insights and streamlining BI workflows, Insights Pro takes on a new approach. Rather than contextualizing data using data dictionary, it leverages the power of LLMs for data dictionary analysis and prompt engineering, thus offering:

• Versatility – Ensures superior data and domain-agnostic performance
• Contextuality – Comes with an advanced data dictionary that understands column definitions and contexts based on session conversations
• Scalability – Spans across different user and verticals

This democratizes access to data-driven insights, reducing the dependency on dedicated analysts. Whether it’s the CEO, Supply Chain Manager, or Plant Manager, they can directly interact with the platform to get the relevant insights on time and as needed.

Empowering Data-Driven Decision-Making | Applications across various industries

Logistics and Warehousing: AI powered BI solutions can assist in optimizing warehouse operations by analyzing shipment punctuality, fill rates, and comparing warehouse locations. It identifies areas for improvement, determines average rates, and pinpoints critical influencing factors to enhance efficiency and streamline processes.

Transportation: Transportation companies can evaluate carrier performance, identify reasons for performance disparities, and assess overall carrier efficiency. It provides insights into performance gaps, uncovers the causes of delays, and supports informed decision-making to optimize transportation networks.

Supply Chain Management: AI powered BI solution empowers supply chain leaders to identify bottlenecks, such as plants with the longest loading times, compare location efficiency, and uncover factors impacting efficiency. It guides leaders towards clarity and success in navigating the complexities of supply chain operations, facilitating data-driven optimization strategies.

Business Intelligence and Analytics: Analysts are equipped with a comprehensive view of key metrics across various domains, such as shipments across carriers, order-to-delivery times, and modeling to understand influencing factors. It bridges data gaps, simplifies complexities, and offers clarity in data analysis, enabling analysts to derive actionable insights and drive data-informed decision-making.

Undeniably, empowering BI through AI can only be achieved by knocking off time-consuming bottlenecks that hinder data-to-insights conversion.

Tiger Analytics’ Insights Pro also goes a long way to combat other challenges that Generative AI has been associated with at an enterprise level. For instance, it ensures that data Security concerns are uploaded to the GPT server as data dictionaries. It also delivers an up-to-date data dictionary so that new business terms shouldn’t be manually defined in the current session.

Looking ahead, NLP and GenAI-powered solutions will break down barriers to data accessibility, automate the discovery of hidden patterns empowering users across organizations to leverage data insights through natural language interactions. By embracing solutions like Insights Pro, businesses can unlock the value of their data, drive innovation, and shape a future where data-driven insights are accessible to all.

The post Empowering BI through GenAI: How to address data-to-insights’ biggest bottlenecks appeared first on Tiger Analytics.

Chat Intelligence: When Chat Meets GenAI and NLP

CHAT INTELLIGENCE refers to our specialized technology and solutions that leverage NLP and GenAI capabilities. At its core, Chat Intelligence encompasses the use of advanced AI-driven algorithms to enhance chat and messaging systems. These systems can understand, interpret, and generate human-like text, based on natural language input, resulting in more sophisticated and valuable user interactions.

Chat intelligence helps drive incremental business opportunities by identifying:

• New leads for businesses
• Signals from existing customers for additional business opportunities
• Potential customer dissonance triggering retention measures
• Themes for personalized marketing campaigns
• Upsell or Cross-sell opportunities
• Indicators or patterns that lead to fraud
• Customer retention strategies
• Customer sentiments

From Chat Conversations to Business Insights

For businesses aiming to integrate chat intelligence into their operations, the significance of chat mining cannot be overstated. Chat mining, a fundamental aspect of chat intelligence, entails the extraction of valuable insights from chat data. This process involves analyzing text conversations to decipher customer preferences, behaviors, and sentiments, utilizing the extensive data generated from interactions between customers and chatbots or virtual assistants. By converting this data into actionable intelligence, chat mining becomes a critical tool for businesses focused on enhancing customer experience, optimizing operations, and making informed strategic decisions.

Despite its potential, chat mining faces several challenges, particularly when relying on traditional NLP techniques:

• Limited Contextual Understanding: Traditional approaches like TF-IDF and Word2Vec for feature extraction often struggle to grasp the full context of conversations. This can lead to misunderstandings of customer intent and sentiment, impacting the quality of insights derived from chat data.
• High Computational Requirements: Processing and analyzing large volumes of chat data require significant computational resources. Traditional models, while effective for simpler tasks, can become inefficient and costly at scale.
• Evolving Language and Slang: The dynamic nature of language, including the use of slang and new expressions in chat interactions, poses a challenge for static models that are not continuously updated.

Overcoming Challenges with Deep Learning and Ablation Analysis

To address these challenges, there has been a shift towards leveraging the power of deep learning. At Tiger Analytics we use models like the Universal Sentence Encoder (USE) and Bidirectional Encoder Representations from Transformers (BERT). These iterations represent a significant departure from traditional approaches, offering enhanced contextual understanding and reduced computational burdens.

• Deep Learning Iteration-1 (USE Embeddings + Classifier): The first iteration involves using USE embeddings, which provide a more nuanced capture of semantic information in chat conversations. This approach marks an improvement over TF-IDF by incorporating a broader context.
• Deep Learning Iteration-2 (Fine-tuned BERT Model): The second iteration advances further with the adoption of a fine-tuned BERT model. BERT’s ability to understand the bidirectional context of words in sentences significantly enhances the model’s performance in chat mining tasks.

The Crucial Role of Ablation Analysis

Ablation analysis is a methodical approach to improving chat intelligence systems by systematically removing components, such as layers, neurons, or specific features, to study their impact on the model’s performance. This process helps identify which elements are crucial for the success of the model and which might be redundant or detrimental. The analysis provides insights into how different NLP and AI techniques contribute to the system’s ability to understand and generate language, offering a deeper understanding of the underlying mechanisms.

Ablation analysis becomes particularly valuable in refining deep learning models for chat intelligence. By systematically removing or modifying components of these complex models, researchers and developers can:

• Identify Key Features: Determine which features or model components are most influential in understanding and generating chat-based interactions.
• Optimize Model Performance: Enhance the accuracy and efficiency of chat intelligence systems by focusing on essential elements.
• Reduce Computational Costs: Eliminate unnecessary or less impactful components, thereby streamlining the model for better scalability and reduced operational expenses.

Ablation Analysis illustrated through a series of examples

In the first example “Hi, I am considering moving all my accounts held at an outside firm to your firm.”, the indication of the movement of money from external firms is clear and all the three models are able to pick up the signal of an incoming transfer.

In the second example, “Hello, I am considering moving my account to a different firm.”, the TF-IDF model and the USE embeddings-based model were not able to understand the nuances of the sentence. These are the typical false positives that the model struggled to differentiate:

In the third example, “May I get some help. I am looking to open a new account and start contributing to it.”, the TF-IDF and USE model’s output probabilities are below the threshold and hence are lost opportunities. However, the BERT model’s fine-tuning helps rightly identify this as a valid lead. This leads to a higher volume of leads and minimizes missed opportunities.

The journey towards achieving excellence in Chat Intelligence is both challenging and rewarding. At Tiger Analytics, we are committed to leveraging the latest advancements in NLP and AI to offer solutions that meet the unique needs of our clients. Our expertise in chat mining and the strategic application of deep learning models and ablation analysis have enabled us to unlock new levels of efficiency, insight, and customer engagement. As we continue to innovate and explore the vast potential of chat intelligence, we invite you to delve deeper into our findings and methodologies.

For a more comprehensive understanding of we’ve used ablation analysis and fine-tuned BERT models to build a help extract chat intelligence from conversational data, read our whitepaper- How NLP and Gen AI are helping businesses derive strategic insights from chat conversations

The post Turning Conversational Data into Chat Intelligence with Ablation Analysis appeared first on Tiger Analytics.

Fast forward a few months later, and Generative AI has already begun taking over the narrative where digital transformation is concerned.

Today there’s an exciting opportunity for enterprises to augment Discriminative AI with Generative AI. Prompt engineering and fine-tuning of foundational Large Language Models (LLMs) with enterprise data can enable insurance companies to reinvent most parts of their value chain.

NLP + Generative AI = The Barbenheimer effect?

It’s important to note that over time NLP has progressed from simply understanding and analyzing text to generating intelligent and contextually appropriate responses, summaries, etc. Traditionally, it focused on tasks such as text classification, sentiment analysis, and information extraction.

Traditional NLP and Generative AI, each on their own, are capable of providing immense value to insurance companies. But by combining forces, Generative AI and NLP have now enabled machines to create human-like text and generate meaningful responses. Together, they can empower insurance teams to tackle foundational and complex tasks with newfound intelligence.

Enterprises have a huge opportunity to customize the available foundational models to realize value from a broad range of capabilities that Generative AI offers.

At Tiger Analytics, we use the following approaches based on our clients’ use cases:

• Prompt Engineering using Enterprise-grade Generative Models – e.g.: Chain of thought, zero-shot examples, few-shot examples
• Prompt tuning using any of the open-source LLM models – e.g.: Prompt tuning, Prefix tuning, and PEFT
• Fine-tuning an open-source LLM model for a specific task – e.g.: Causal Language Modeling (CLM) and Masked Language Modeling (MLM) on objectives like summarization, code generation, etc.
• Building a foundational LLM model with multiple tasks capability using RLHF loop.

4 Process Wins for the Insurance Industry

We’ve worked on Generative AI projects involving search and summarization, description generation, and next-gen chatbots for various clients. Here are four processes that Generative AI + NLP can help transform:

Process Win #1: Pricing and Underwriting

With automated information gathering and data entry, insurers can now expedite the search process, leading to faster and more accurate access to crucial data points — and it’s all thanks to Generative AI.

Furthermore, Generative AI’s intelligent search and summarization capabilities enhance risk assessment by swiftly analyzing vast amounts of information and extracting key insights. This streamlines the underwriting process while improving decision-making accuracy.

Generative AI also enables automated documentation through description generation, reducing the need for manual report writing, and thereby reducing the risk of human errors. As a result, insurers can make better pricing and underwriting decisions.

Process Win #2: Claims Processing

Traditionally, adjusters would spend hours manually reviewing adjuster notes and other supporting documents, resulting in delays and potential claims leakage. However, with Generative AI, the game changes. Through automated information gathering and data entry, insurers can leverage faster search capabilities to access relevant information swiftly. Market studies indicate that Generative AI can reduce the time spent reviewing claim files to less than an hour.

Generative AI’s intelligent search and summarization capabilities also go a long way to enable adjusters to extract and analyze key insights, minimizing the risk of claims leakage. This, coupled with a sophisticated Next Best Action (NBA) model, helps adjusters make settlement decisions quickly and scientifically. The impact is profound — insurers can now streamline claims processing, ultimately reshaping the claims landscape in the insurance industry.

Process Win #3: Contact Center Operations

Generative AI can help overcome many current challenges such as the limited conversational abilities of chatbots and the need for agents to navigate multiple systems and documents. With the implementation of a conversational agent, a major portion of customer queries can be self-served, resulting in reduced resolution times. Even more complex queries can be seamlessly routed to agents, ensuring personalized customer experiences.

Also, Generative AI can empower agents with intelligent search and summarization capabilities, making sure they can access relevant information and deliver accurate responses.

Process Win #4: Marketing and Distribution

Traditional chatbots often struggle to provide natural language conversations, hindering their ability to answer product queries. But by working with Generative AI, the effectiveness of sales agents can be improved, equipping them with quick access to relevant information and enabling them to provide accurate and personalized responses. Generative AI can help address critical marketing and distribution challenges for enhanced customer engagement.

Generative AI facilitates better explanations of policies and products through automated description generation, reduces administrative burden, and frees up valuable time for sales agents. Ultimately, insurers are equipped with the technology firepower to deliver consistent and contextual customer journeys across touchpoints, creating tailored experiences that align with individual preferences.

Just as the iPhone revolutionized human interaction with technology, Generative AI is poised to revolutionize the insurance landscape by simplifying and accelerating the digital transformation journey for insurers. Paired with NLP capabilities, it will help change the way insurers automate and expedite key workflows, ranging from risk assessment to claims processing and underwriting. Our brave new world is here.

The post AI-Powered Insurance Wins: Unlocking Process Efficiencies with NLP and Generative AI appeared first on Tiger Analytics.

For an insurance company, however, complying with this mandate can involve a lot of intricacies. How to make sure this process is not resource-intensive? How to take into account all the data available? These were problems a large insurer engaged with Tiger Analytics to solve.

They had a rules engine with manually created rules, but it led to lower approval rates. The question was, can this process be augmented with Artificial Intelligence from data?

In most large enterprises today, we now have multiple sources of enterprise data – ranging from the Data Warehouse, Salesforce, PDF document dumps, etc. but the challenge is to pool all the data together. In this case, to enrich the set of data elements, our team used standard database connections to pull the data and collate it. Structured and unstructured data were present across different buckets:

• Client demographics such as age, income, and investment experience
• Details about the finance professional making the sale, such as tenure and history of past applications
• Details about the application, including investment objective, risk tolerance, expected portfolio composition, and suitability recommendation rationale.

Some of this data are structured fields, but others are free text which is complex to analyze.

To comply with regulators, explainable features had to be extracted from free text fields like the recommendation rationale. This is where NLP came in, helping extract structured information from free text for use in a subsequent model. Similarly, other NLP techniques were also used to extract information from text, resulting in the creation of features such as product suitability for the buyer and product recommendation by the finance professional.

The next step was to build a model which could classify a particular application as auto-approved or requiring follow-up. Several considerations could be indicative of approval, including:

• If the finance professional involved was tenured, there’s less likely to be a need for follow-up.
• If the source of the buyer’s funds was from multiple sources, follow-up is more likely.
• The more assets the buyer holds, the less likely follow-up is needed.

Classification models were built using data elements and extracted features representing these considerations. The methods included logistic regression – a statistical method that predicts the probability that the application should be approved; and tree-based models such as decision trees and random forests, which look to map combinations of the features to being auto-approved or not. While implementing a model with material consequences for customers’ money, it’s critical to ensure the model is well-tested on unseen scenarios. Therefore, the model was validated on multiple samples it had not been trained on. Additional business rules were also identified through the course of analysis.

The final system for approval combined both the existing rule engine and the analysis outputs. The AI model and identified business rules were deployed as a layer on top of the rule engine. Model outputs were explained using state-of-the-art methods such as Shapley Additive Explanations (SHAP). With this system, the company saw an improvement in auto-renewals by 29 percentage points (from 43% to 72). Our proprietary accelerators for solution development, such as TigerML and Blueprints, enabled the quick realization of value, helping wrangle the data and apply Machine Learning methods rigorously.

For a large enterprise, the low-hanging fruit for the application of AI is in augmenting rules. Another area is using NLP to extract information from the vast amounts of unstructured data available. Organizations that are part of a regulated industry (like financial services) must ensure that the models are explainable, using explainable features and methods like SHAP. This can unlock efficiency in a range of regulatory processes and have a direct impact on the bottom line.

The post How Insurance companies are using NLP to streamline application approval appeared first on Tiger Analytics.