How to scrape data?

Scraping data is such a useful tool as it means that you can grab bits of information from various websites and it is in a form that you can easily analyse.

Paul Bradshaw, Data journalist, said in a article with Journalism.co.uk that:

 “I love scraping: it is both a great time-saver, and a great source of stories no one else has.”

So Data Queen has decided to experienced for her self how easy it is to scrape. I scraped a few lines from parliamentary website and found out that it is not only easy but it is also incredibly useful.

It was a joy to use and I would most certainly use it again. So I have to share m,y new found knowledge with you and here goes.

I decided to look at the financial interests and payments made to MPs. These payments are made from third parties for the time it may take for an MP to do a talk or to attend an event where there experience and expertise is needed.

I started with MP Diane Abbot.

First I copied and pasted the URL into outwit hub.

I used Outwit hub as it was very easy and not a difficult piece of software to use and scrape data from.

Image

Then I looked for the information I wanted. So I wanted to scrape her name as well as the amount of money she was being paid for a particular event.

So I started a new scrape, and than I added in a marker before and a marker after. Now the marker that you put in depends on the information you want to scrape. For example once you have copied and pasted the link you can then scroll down the page and you are able to look for the marker that you require.

If you look before the persons name you will than be able to see the marker you need. In my case the marker before was </div> and the marker after was </h2>  (please see image below).

Image

Once I had the marker before and after in I was able to save the first scrape and execute. Once you have done this you can export the data to an excel format.

Image

Below you can see the events that Diane Abbot went to and the payments she received for them. All good.  Diane Abbott we love you.

Image

Reflection

I enjoyed this, however there were times that it was quite a difficult process. Sometimes it involved some trial and error when choosing the correct marker before and after.

However it provided some  useful information and I would highly recommend people use.

What I would recommend is perhaps try and do this on a few different websites. Do not chose a complicated one perhaps like the one I chose as it makes it more difficult to find the marker before and after.

Thanks for reading!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s