CASE STUDY
Streamlining Financial Report Access Globally to Boost Scalability & Performance
Introduction
The document will give an overview of a solution implemented for searching and buying financial reports for any company worldwide with translation in two different languages and display alerts to subscribed users. It gives profound insights into the overall project journey including the challenges, achievements, and blockers.
This document gives an overview into the journey of Insta International Report, a solution unlocking unprecedented access to financial reports for any company worldwide. It offers seamless searching, purchasing, and translation in two languages, alongside timely alerts for subscribing users. We’ll delve into the challenges overcome, milestones achieved, and roadblocks encountered along the way, providing a holistic overview of this transformative project.
Insta International Report: Key Features
- Expanding Reach: Introduce GDN reports to our international proposition, boasting faster turnaround times. This opens doors to previously untapped client segments, fueling significant market growth.
- Automating Efficiency: Embrace automation through API integrations with GDN, Worldbox, and others. This streamlines report fetching, boosting team productivity and freeing up resources for strategic initiatives.
- Future-Proofing BI: Lay the groundwork for our upcoming BI replatforming. Insta International Report’s architecture aligns with the approved Nike tech stack, paving the way for smooth integration and future advancements.
Vendor Landscape:
- GDN (Global Data Network): Provides online reports with swift delivery times.
- World Box: Offers online and offline reports, catering to diverse client needs.
- CCS (Client Care System): Handles billing for purchased reports.
Comprehensive Monitoring:
Insta International Report empowers users with a centralized hub for all subscribed alerts and name list data. This ensures instant access to critical information and fosters informed decision-making.
The Business Challenge
Insta International Report (IIR) empowers users to search and purchase financial reports for any company worldwide. Its core functionalities include report acquisition from vendors like GDN and Worldbox, displaying alerts for subscribed users, and report generation with email notifications. However, the initial architecture presented several challenges:
IIR (Insta International Report) is a platform allowing users to search and purchase financial reports for any company worldwide. It integrates with two vendors, GDN (Global Data Network) and World Box, to provide online and offline reports, and displays subscriber alerts in the monitoring section.
Monitoring: Due to reliance on external data providers, the team lacked control and needed to call 3 different APIs, implement business logic for proper data formatting, and send email notifications (for both alerts and no alerts) to all subscribed users. A single API call could involve displaying up to 10,000 alerts or more. Users can add, edit, delete, or batch upload (up to 1,000 records) alerts.
Reporting: Reports receive data in JSON format from the external service provider. The final report format requires merging another JSON file into the existing one based on specific conditions. Email notifications are sent based on report type, with each type having a unique email body.
Performance Bottlenecks: The current process, involving multiple API calls, authentication, and data formatting, led to significant slowdowns. Key bottlenecks included:
- Fetching data from the external service provider, storing it in IIR database tables, and formatting it for display (often exceeding 1,000 rows).
- Merging data from two separate JSON files for report generation.
- Sending email notifications based on customer profiles (daily/weekly for alerts, report-specific in reporting, each with differing subject and body sections).
- Validating excel file format, length, and data during batch upload (up to 1,000 records per batch).
- Solution Challenges: Each action within the application, from data fetching to email sending, involved multiple API calls, leading to timeouts and difficulty separating alert data for different monitoring types per subscriber. Batch upload validations in both reporting and monitoring also caused timeouts.
Hoonartek’s Solution: To overcome these challenges and achieve a highly scalable, cross-platform application, the team implemented:
- Microservices: Divide functionalities into independent, smaller services for improved performance and agility.
- .NET Core APIs: Leverage the efficiency and cross-platform capabilities of .NET Core APIs.
- Background Services: Handle long-running tasks like email sending and alert storage asynchronously.
- Store Procedures: Return data in a single JSON column and row format for optimized retrieval.
- Schedulers: Automate recurring tasks like notification sending and large-data alert storage.
- Excel Mapper Library: Validate excel file length, columns, and data format, and efficiently read data.
Outcome: This proposed solution effectively addressed the identified challenges, resulting in a highly scalable and cross-platform application.
Facing Scalability Challenges?
Conquer scalability challenges with our proven solution for building highly scalable, cross-platform apps. Contact us now!
Joint Solution Discovery & Implementation Plan
Leveraging the solution, the team aimed to achieve project goals and overcome the mentioned challenges. To implement this plan, they employed the Facade design pattern, which hides system complexities and provides a client interface for accessing the system. Essentially, the HTK team proposed a solution addressing the previously identified challenges.
Development Stack: The backend utilized .NET Core APIs, while the frontend employed Angular technology. Postman served as the tool for API testing. As part of the solution, the team built a database structure tailored to the application’s needs. Initially, they opted for the Entity Core framework for writing database queries. However, timeout errors experienced by frontend users due to entity queries prompted them to seek alternative solutions as followed:
Solution Process:
- Microservices: Each complex module with extensive business logic was converted into a dedicated microservice. Schedulers were created for email notifications and alert data storage, operating via command-line arguments. A central Business Logic Layer (BLS) was established to handle all request processing.
- Authentication and Validation: BLS handled user validation within session middleware and generated tokens for API access. Armed with session ID and token information, BLS directed requests to the appropriate API services.
- Query Optimization: To overcome entity query issues, the team split them into smaller steps. They created stored procedures returning single-row, single-column JSON outputs, enabling efficient retrieval of large datasets within minutes. For batch uploading, a background service was implemented, running tasks asynchronously without user wait times.
- Email Notifications: A master table was designed to store information about notification types, including pre-built minified HTML templates for each type. Cron jobs were then created to automate email notifications based on provided command-line arguments, eliminating reliance on manual triggers.
The open-source Excel Mapper library was chosen for its ease of use, minimal coding requirements, and reduced time investment compared to writing entity queries against datasets provided by the library. File validation was also simplified with the library.
Completed reports offered a Japanese translation option for users. For email notifications, schedulers were again utilized, operating on command-line arguments.
Overall, the proposed solution effectively addressed the initial challenges and paved the way for a highly scalable, cross-platform application.
Outcomes
By embracing the proposed architecture, the HTK team successfully attained several key objectives:
Cross-platform Support
The application seamlessly operates across diverse platforms, maximizing accessibility and user reach.
Performance Optimization
Implemented solutions effectively addressed performance bottlenecks, resulting in a significantly faster and more responsive application.
Enhanced Maintainability
Utilizing microservices architecture simplified code structure and module independence, enhancing maintainability, and facilitating future updates.
These achievements demonstrate the effectiveness of the implemented solution in overcoming initial challenges and delivering a robust, scalable, and future-proof application.
Recommendations
The implemented solution offers several key advantages worth highlighting:
- Open-Source Platform: Built on open-source technologies, the application enjoys cross-platform compatibility, seamlessly running on Windows, Linux, macOS, and other operating systems. This eliminates migration concerns and allows for future integration with new .NET Core features without additional effort.
- Simplified Deployment: The modular architecture facilitates independent deployments for each microservice, streamlining the process and fostering agility in responding to changing needs.
These benefits position the solution as a robust, adaptable, and future-proof choice, well-equipped to evolve alongside technology and user requirements.
