Fun with Scraping - Axiom.ai

Who are we? We are developers. And what do we like to do? We like to automate. Although this statement is slowly losing its value. Now-a-days everyone is a techie and the automation bug has bitten everyone. Axiom.ai is a simple tool that allows browser automation and believe me when I say - it doesn't get better than this.

Here is what we want to do today. We want to get some trending articles about a topic so that we can share it to someone. We will be using dev.to as our target. The agenda is to open dev.to (source), find articles (function) related to a tag and then export them to a sheet (database).

We start by installing the axiom.ai chrome extension from here and creating an account. Once done, click on the browser extension and create a New Automation.

You will be tempted with a lot cool things to do. For now, we will go ahead with a simple blank template.

We will click on Page Interaction as we would like to click on the search bar and find all the posts that have a nodejs tag in them. The login part to the website has been skipped intentionally as it is inline with what we are doing right now.

You will notice axiom allows us to interactively select the page contents just like a normal user would.

Once the element is selected we will type in the text we want to search.

Once done we will tell axiom to click on the search button and navigate to the next page.

On the next page we want to select the title, author, date of publish, reactions and time to read and save.

We will use Get Data from a webpage feature to do this.

We will select each one of these elements from the page and tell axiom to save it as a table. Here is a nice preview of the selected things. While selecting the page elements that you want to scraped, make sure to select multiple of them so that axiom is able to figure what exactly you are trying to do.

Once done we will use another feature called Export to CSV to save it into a normal csv for now. The all-interaction-data flag allows us to work with data from a previous step. We can also tell axiom how many such elements we are looking to scrape even. 100 for now.

Axiom also allows us to export to google sheets or even save it to a database using a webhook to a hosted service.

And that is it. Here is what we have done so far in action.