This is a remote position.
About the Role
We are looking for a highly skilled Senior Social Media Scraping & Automation Engineer to build a scalable platform for collecting and processing publicly available data from social media platforms and other web sources. You will design reliable scraping infrastructure, browser automation, API integrations, and data pipelines capable of processing large volumes of data efficiently.
Key Responsibilities
· Design and develop scalable web scraping solutions for social media platforms and websites.
· Build browser automation using Playwright, Selenium, or Puppeteer.
· Integrate official platform APIs wherever available.
· Develop systems to collect publicly available information, including user/business profiles, posts, comments, public engagement metrics, hashtags, keywords, company information, and public contact information.
· Build scheduled data collection and monitoring systems.
· Design distributed scraping architecture capable of handling high-volume workloads.
· Implement proxy rotation, session management, cookie handling, browser fingerprinting basics, and rate-limiting strategies.
· Optimize scraper performance, reliability, and fault tolerance.
· Develop ETL pipelines to clean, validate, deduplicate, and enrich collected data.
· Deploy and maintain scraping services on cloud infrastructure.
· Monitor scraper health with logging, alerting, and automated recovery.
· Collaborate with product, data, and engineering teams.
· Maintain technical documentation and follow engineering best practices.
Required Technical Skills
Programming
· Python (Expert)
· JavaScript / TypeScript
· SQL
· Git
Scraping & Automation
· Playwright (Preferred)
· Selenium
· Scrapy
· BeautifulSoup
· Requests / HTTPX
· Asyncio & Multiprocessing
Social Media APIs
· X (Twitter) API
· LinkedIn API
· Meta Graph API (Facebook & Instagram)
· Reddit API
· YouTube Data API
· Telegram API
· Discord API
Web Technologies
· HTML
· CSS
· JavaScript
· GraphQL
· AJAX
· WebSockets
· Chrome Developer Tools
Databases
· PostgreSQL
· MySQL
· MongoDB
· Redis
Cloud & Infrastructure
· Docker
· Kubernetes (Preferred)
· AWS, Azure, or GCP
· Linux
Data Engineering
· Pandas
· ETL Pipelines
· Apache Airflow (Preferred)
· Kafka or RabbitMQ
Requirements
Preferred Qualifications
· Experience building distributed scraping infrastructure.
· Knowledge of proxy management, browser fingerprinting, and session handling.
· Experience with OCR and document processing.
· Experience integrating AI/LLMs into automation workflows.
· Experience building monitoring dashboards and alerting systems.
Qualifications
· Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
· 4-8+ years of experience in Python development and web automation.
· Strong problem-solving and system design skills.
· Excellent communication and documentation skills.