Project Overview
In this case study, I explore the challenges of building a modern listings website that aggregates data from multiple sources and presents it in a user-friendly and visually appealing manner. My goal was to create a platform capable of handling diverse data formats, ensuring real-time updates, and providing a seamless user experience. This project allowed me to combine my passion for data scraping with frontend development, utilizing Python for data aggregation and Vue.js for the user interface. A preview of the app is live here.
Tech Stack
- Python (Scrapy, FastAPI)
- Vue.js
- PostgreSQL
Key Challenges
- Data Fragmentation: Data was spread across multiple websites, each with a unique HTML structure, making scraping and consolidation difficult.
- Data Consistency: Ensuring that the scraped data was clean, consistent, and free of duplicates or errors was crucial.
- User Experience: The frontend needed to be intuitive, responsive, and visually appealing to attract and retain users.
Solutions
Data Scraping with Python:
- Leveraged the Scrapy library to asynchronously scrape data from multiple sources. This included implementing data cleaning, validation, and pipelining.
- Set up a scheduler using cron jobs to periodically scrape data and update the database in real-time.
Backend Development:
- Built a backend API using FastAPI to serve the scraped data.
- Designed a database model for efficient storage and retrieval of listings.
Frontend Development with Vue.js:
- Created a responsive and interactive frontend using Vue.js to display listings in a user-friendly way.
- Implemented features such as search, filtering, and pagination to help users easily find what they were looking for.
- Utilized Vuex for state management to ensure a smooth and consistent user experience.
Lessons Learned
- Gained hands-on experience in UI/UX design using Figma, learning to create user-centric interfaces that enhance usability and engagement.
- Developed proficiency in frontend development with Vue.js, including state management, component-based architecture, and responsive design.
- Mastered building scalable data scraping pipelines, ensuring efficient data collection and processing from diverse sources.
- Learned the importance of flexibility and adaptability when scraping data from websites with varying HTML structures, which required creative problem-solving and robust error handling.
Attachments
Look Estate Home Page
Look Estate Detail Page
Link to the project: Look Estate