Overview
Built an automated data pipeline that fetches vendor volume data from the HPSCAT API and uploads JSON snapshots to SharePoint Online for internal reporting and operations teams. The pipeline handles API unreliability with intelligent retry logic and runs unattended on a schedule for both single-vendor and multi-vendor workflows.
Key Features
- Configurable API integration with date offset support for historical data pulls
- Intelligent retry logic (3 attempts, 10-minute delays) for rate-limited APIs
- Detection of rate limit responses embedded in HTTP 200 bodies
- JSON data preservation with 100-level depth for complex nested structures
- UTF-8 no BOM file output with dated filenames
- Microsoft Graph API upload to SharePoint with dynamic site/drive discovery
- Automatic folder creation during upload operations
- Separate single-vendor daily pulls and multi-vendor monthly batch processing
- Comprehensive error logging with full Graph API response details
Technical Highlights
- OAuth 2.0 client credentials flow for Microsoft Graph authentication
- Azure AD app registration with Sites.ReadWrite.All permissions
- PowerShell data file (.psd1) for environment-specific configuration
- Dynamic SharePoint path parsing for /sites/SiteName/Library/Folder structures
- Hardcoded Site/Drive ID fallback for faster execution on known targets
- Exponential backoff retry mechanism for API resilience
- Scheduled execution via Windows Task Scheduler