We need to have website with the following features.
To allow ARDC to search across 35+ instant access dialers (vendors) covering 200 countries, domestic UK national and mobile rates – inclusive of recent VOIP offerings. A list of vendors that the we need to gather data from is provided in Appendix A. An example of an implementation can be found on [url removed, login to view]
2. Generating Searchable Information
This is the most complex part of the implementation and can be broken down into three phases.
1. Data Gathered from website
2. Clean-up an validation on data
3. Write to DB
4. Error Reporting
è Error Reporting.
2.1 Data Gathering ( It will be automatic on a defined time interval but there should be the option to trigger this on-demand.)
Data can be gathered in one of two ways, either through a defined feed provided by the vendor (e.g. CSV, Deliminated File, XML, RSS) or, as will be the case for most of the vendors, through a HTML “scrapping” process. It should be noted that the websites don't all display data in the same format.
Data from the site needs to be mapped into Our Data Model defined below.
Our Data Model:
Field Example Mandatory
UID 1 Y
Vendor Justdial Y
Country Spain Y
LandLine Cost 4 Y
Mobile Cost 10 N
Fixed Line Number 0870 123 3400 Y
Mobile Number 0798 898 8899 N
Time Restriction 7am to 7pm GMT N
Website URL [url removed, login to view] Y
A UID corresponds to a unique vendor/country pair
Issues with doing this real-time? Or otherwise daily overnight (GMT) batch or hourly?
3.1 Use Cases
User selects “calls” tab which opens up a new page below and specific search dialog.
User can then query the DB in one of two ways:
Free Text Search for country / type (type = mobile or landline)
Select Country from alphabetic list.
A match occurs when a country / type has been identified.
For a given match the User will be shown a table as shown below which will be populated with all matched vendors listed in Appendix A.
Access Number Vendor LandLine Price Mobile Price Problem?
Subscribe? If a user subscribes to a match (and is configured accordingly in their profile) they will receive an alert via email and/or SMS when the price of the match drops below the cheapest shown at the time of the search (note: should charge for this if SMS)
Problem? – if the user selects this then the event needs to be registered in our DB. It should also dynamically open up a small comments text box (to the right or below) where the user can select from a predefined list of problems (see xyz) and / or enter in a short comment in a free text box. The I.P address of the person who logged the event should also be stored.
To reconfirm there is no need for a GUI.
The systems needs to crawl and grab information from the websites e.g.
from the HTML pages on this site:
[url removed, login to view]