2020-10-05 08:24:33 +00:00
|
|
|
## 1- INTRODUCTION
|
|
|
|
|
|
|
|
|
|
Scraper is a project made in Golang to scrap different types of data about sports, such as :
|
|
|
|
|
- all news about sports
|
|
|
|
|
- schedules and scores [coming soon]
|
|
|
|
|
- team staff [coming soon]
|
|
|
|
|
- details about players [coming soon]
|
|
|
|
|
- tv schedule [coming soon]
|
|
|
|
|
|
|
|
|
|
For that, different technologies have been used :
|
|
|
|
|
- `Golang` (with `go routines`, `goquery`, `gofeed`, `net/proxy`)
|
|
|
|
|
- `PostgreSQL` (used to store data)
|
|
|
|
|
- `InfluxDB` (used to store details about each program execution) [coming soon]
|
|
|
|
|
|
|
|
|
|
Several websites are stored in order to gather different types of data :
|
|
|
|
|
- eurosport.fr
|
|
|
|
|
- rugbyrama.fr
|
|
|
|
|
- fftt.com
|
|
|
|
|
- footao.tv [coming soon]
|
|
|
|
|
- football.fr [coming soon]
|
|
|
|
|
- football365.fr [coming soon]
|
|
|
|
|
- football-direct.com [coming soon]
|
|
|
|
|
- footmercato.net
|
|
|
|
|
- lequipe.fr
|
|
|
|
|
- matchendirect.fr [coming soon]
|
|
|
|
|
- programme-television.org [coming soon]
|
|
|
|
|
- transfermarkt.fr [coming soon]
|
|
|
|
|
|
|
|
|
|
All these data are collected in nonprofit purpose for `1bet.fr`, a website made for free sports
|
|
|
|
|
predictions between friends.
|
|
|
|
|
|
|
|
|
|
I decline any responsibility about your eventual usages of this project.
|
|
|
|
|
|
|
|
|
|
|
2020-10-05 08:26:18 +00:00
|
|
|
## 2- DEPLOYMENT
|
2020-10-05 08:24:33 +00:00
|
|
|
|
|
|
|
|
The deployment is very simple as the binary `scraper` can be used directly.
|
|
|
|
|
|
|
|
|
|
A PostgresSQL database is needed for this program, as well as some environment variables,
|
|
|
|
|
all clearly listed in `postgres.go`.
|
