Data Acquisition in Retail - Adaptive Data Collection

Introduction

For Intelligence Node to provide its customers with real-time, high-velocity, and accurate data means crawling millions of web pages every day to extract the right information. But data acquisition at such a large scale and at affordable costs is not possible, manually. It is a rigorous process and it comes with its own challenges. To address these challenges, Intelligence Node’s analytics and data science team has developed strategies through advanced analytics and continuous R&D.

In this part of the ‘Alpha Capture in Digital Commerce series’, we will explore the challenges of data acquisition in retail and discuss data science applications to solve these challenges.

Adaptive Crawling

Adaptive crawling consists of 2 components:

The Elegant Middleware: Smart Proxy

Intelligence Node’s team of data scientists has worked on developing intelligent, automated strategies to overcome crawling challenges such as high costs, labor intensiveness, and low success rates.

• Builds a recipe (plan) for the target from the available strategies
• Tries to minimize it based on:
• Price
• Success rate
• Speed

Some of the strategies are

• Election decision of a certain IP address pool
• By using mobile/residential IPs
• By using different user-agents
• With a custom-developed browser (cluster)
• By sending special headers/cookies
• Using anti-blocker [Anti-PerimeterX] Strategies

The Heavy Lifting: Parsing

Auto Parsing

• The team utilizes a custom-tuned transformer-encoder-based network (similar to BERT) which converts webpage to text for information retrieval of generic information available on product pages such as price, title, description, and image URLs.
• The network is layout aware and utilizes CSS properties of elements to extract text representations of HTML without rendering it as opposed to the Selenium-based extraction method.
• The network is capable of extracting information from nested tables and complex textual structures as the model have an understanding of both language and HTML DOM

Visual Parsing

Another way of information extraction from web pages or PDFs/screenshots is through Visual Scraping. Often when crawling is not an option, the analytics and data science team use a custom-built visual, AI-based crawling solution.

Details

• For external sources where crawling is not permissible, the team uses visual AI-based crawling solution

• The team uses Object Detection using Yolo (CNN-based) architecture for precise identification of product pages into objects of interest such as title, price, information, and image area.

• The team sends pdfs/images/videos to get textual information by attaching OCR Network at the end of this hybrid Architecture.

   

Example

Tech Stack

The team uses the below tech stack to build the anti-blocker technology widely used by Intelligence Node:

Linux (Ubuntu), a default choice for servers, acts as our base OS, helping us deploy our applications. We use Python to develop our ML model as it supports most of the libraries and is easy to use. Pytorch, an open source machine learning framework based on the torch library, is a preferred choice for research prototyping to model building and training.

Although similar to TensorFlow, Pytorch is faster and is useful when developing models from scratch. We use FastAPI for API endpoints and for maintenance and service. FastAPI is a web framework that allows the model to be accessible from everywhere. We moved from Flask to FastAPI for its additional benefits like simple syntax, extremely fast framework, asynchronous requests, better query handling, and world-class documentation.

Lastly, Docker, a containerization platform, allows us to bundle all of the above into a container that can be deployed easily across different platforms and environments. Kubernetes allows us to automatically orchestrate, scale, and manage these containerized applications  to handle the load, on autopilot - if the load is heavy it scales up to handle the extra load and vice versa.

Conclusion

In the digital age of retail, giants like Amazon are leveraging advanced data analytics and pricing engines to review the prices of millions of products every few minutes. And to compete with this level of sophistication and offer competitive pricing, real-time price monitoring, assortment, and personalized experiences to today’s comparison shoppers, AI-driven data analytics is a must. Data acquisition through competitor website crawling has no alternative. As the retail industry becomes more real-time and fierce, the velocity, variety, and volume of data will need to keep upgrading at the same rate. Through these data acquisition innovations developed by the team, Intelligence Node aims to constantly provide the most accurate and comprehensive data to its clients while also sharing its analytical abilities with data analytics enthusiasts everywhere.

#graphicdesign #marketing #instagram #dev #aforisma #frasi #softwareengineering #pensierodelgiorno #frontend #programmer #radio #pensiero #wireframe #clubdelcomic