samuel
/
scraper
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6 Commits 1 Branch 0 Tags 29 MiB
Go 100%
Go to file
Samuel Campos 748bbb8048 Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
match Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
news New function updateSchedule 2020-10-19 11:26:23 +02:00
postgres Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
requests New function updateSchedule 2020-10-19 11:26:23 +02:00
utils Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
.gitignore Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
go.mod New function updateSchedule 2020-10-19 11:26:23 +02:00
go.sum New function updateSchedule 2020-10-19 11:26:23 +02:00
main.go Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00
readme.md New function updateSchedule 2020-10-19 11:26:23 +02:00
scraper Some improvments in postgres client with match and team objects + add some utils functions 2020-12-02 17:58:49 +01:00

readme.md

1- INTRODUCTION

Scraper is a project made in Golang to scrap different types of data about sports, such as :

  • all news about sports
  • schedules and scores [coming soon]
  • team staff [coming soon]
  • details about players [coming soon]
  • tv schedule [coming soon]

For that, different technologies have been used :

  • Golang (with go routines, goquery, gofeed, net/proxy)
  • PostgreSQL (used to store data)
  • InfluxDB (used to store details about each program execution) [coming soon]

Several websites are stored in order to gather different types of data :

  • eurosport.fr
  • rugbyrama.fr
  • fftt.com
  • footao.tv [coming soon]
  • football.fr [coming soon]
  • football365.fr [coming soon]
  • football-direct.com [coming soon]
  • footmercato.net
  • lequipe.fr
  • matchendirect.fr [coming soon]
  • programme-television.org [coming soon]
  • transfermarkt.fr [coming soon]

All these data are collected in nonprofit purpose for 1bet.fr, a website made for free sports predictions between friends.

I decline any responsibility about your eventual usages of this project.

## 2- DEPLOYMENT

The deployment is very simple as the binary scraper can be used directly.

A PostgresSQL database is needed for this program, as well as some environment variables, all clearly listed in postgres.go.