Write a string fuzzy matching algorithm script
- Status: Closed
- Prize: $500
- Entries Received: 11
- Winner: x3x7
Contest Brief
Problem:
We have 2 datasets with lists of organizations. they need to be matched.
List-1 contains 4 columns:
1. organization
2. countryCode
3. regionName
4. cityName
List-2 contains 6 columns:
1. ID
2. organization
3. countryCode
4. regionName
5. cityName
6. website-url
We need to match both list on column “organization”
Desired End Result:
A script/program that uses List-1.csv and List-2.csv and outputs List-3.csv
List-3.csv is a new list with 4 columns
1. ID (from list 2)
2. organization (from list 1)
3. organization (from list 2)
3. matching score (0-100)
notes:
You can use all available columns to get better matching
Feel free to use third-party sources to get better matching
The person with the highest amount of matches with a high score wins the contest
Recommended Skills
Public Clarification Board
-
Contest Holder - 11 years ago
Hi all, we have selected a winner: x3x7
Below is a list of entries and relevant matches, as you can see x3x7 has not only the most matches, he (or she?) also provided good documentation, screenshots and iterations on the results.
Thank you all for your effort.
results:
1 x3x7 898
2 Влад Миронов 787
3 Marouane BENMEIDA 560
4 saniokazzz 550
5 Louis Newstrom 500
6 jatindhood 460
7 Marek Sobczuk (Wro) 430
8 Željko Kraljević 150
9 Christy Zach ?
10 Svet Batoev (svet63) ?
11 princewhin ?
12 uniquecode1 ?- 11 years ago
-
vw1249172vw
- 11 years ago
I suggest that the participants say what is the big O of their solutions, since O(n²) is not realistic.
- 11 years ago
View 22 more messages-
x3x7
- 11 years ago
Actually, while preprocessing List-2 helps, the only important issue it to have the List-1 block in cache (the innermost loop), I can then run List-2 fully, no need for extra complications...
- 11 years ago
-
x3x7
- 11 years ago
I've read now the memory calculations you made. Perhaps needles to explain, block-wise for List-1 the mem transfer is reduced 10k times (or how big the block is). The second observation is that in my case the important data needed for a row is a fixed 16 bytes since most pairs fail at this 1st level test. If that passes then I need the extra ~48 bytes for a proper comparison.
- 11 years ago
-
vw1249172vw
- 11 years ago
Hi can we know if our results file are close or far from what expected ?
- 11 years ago
-
vw1249172vw
- 11 years ago
The request is made so if the response is positive, I spend more time to:
1) Make tests in environment close to real.
2) Add the cityName in matching to get less duplicates, and this need ping/pong with the database, I need to check if it will be not a heavy procedure.- 11 years ago
-
Microlent Systems
- 11 years ago
Hello Edie, Please let me know in case if you want me to make any corrections in my script. Thanks much
- 11 years ago
-
mattsrinc
- 11 years ago
Hi, so you don't require for example VBA script in Excel? I would solve your problem in R (in fact will start ASAP). Of course I can output final data in your List-3.csv.
- 11 years ago
-
codeteam17
- 11 years ago
Can you provide clearer definition of what means for two organisation names to be matched or is it up to contestants to define that? Can you upload the big input files somewhere, so we can test with them after we make the system work for this small input? Is there execution time limit? What is the memory footprint limit of the program?
- 11 years ago
-
codeteam17
- 11 years ago
That would be helpful. What about time limit? How long software can load and process the entries before you consider it failed?
- 11 years ago
-
Contest Holder - 11 years ago
There is no time limit defined, but 24 hours seems reasonable
- 11 years ago
-
ergo1wish
- 11 years ago
Hello. I have one question regarding the matching.
If you have two entries:
one in List-1 - with company name Google
second in List-2 - with company name Google
But all the other informations like:
- countryCode
- regionName
- cityName
are different (so lets say google in America and Europe)
Should this two entries be matched with a 100% or a 0% similarity or something in between?- 11 years ago
-
ergo1wish
- 11 years ago
Thanks, and one more follow up questions: if you have two companies like
1. company name: Fashion designs (or something like that)
2. company name: Clothes manufacturer (or something like that)
All the information for this company's are different (the city, region etc.). The names are also very different, but in fact this two companies do a very similar job (making clothes). Again do you want to match this companies by 0%, 100% or something in between (higher or lower then 50%).
Basically I'm interested: are you matching companies by the things they do, or just by their name, location etc.- 11 years ago
-
Contest Holder - 11 years ago
no problem.
in you example it would be 0%.
match on just the name and location.
We want to match the exact same companies from both list- 11 years ago
-
x3x7
- 11 years ago
I'm new here so rules for contests are not quite clear for me... Is it important to offer a solution very quickly?
- 11 years ago
-
svet63
- 11 years ago
Just submitted 5 min ago.
- 11 years ago
-
Contest Holder - 11 years ago
sorry, i ment oct 17
- 11 years ago
-
x3x7
- 11 years ago
I had more questions but wasn't a good idea to press Enter to separate them with new-lines. Q2: Since only a result list is submitted, how are you going to evaluate the quality of matching? Q3: How important is speed?
- 11 years ago
-
Contest Holder - 11 years ago
we will count the result with the most matches, or high matching scores. also we will look at the possibility to apply it to the master set that has 4 million rows.
We want people be creative and help us find the best solution.- 11 years ago
-
svet63
- 11 years ago
Thanks for the feedback. I'll submit my sample later tonight.
- 11 years ago
-
svet63
- 11 years ago
These 2 sample lists have to few matches. Plus the number of entries differ. As the #1 ID is in list 2, does that mean that we only match the entries in 2 against 1? What output do you expect considering that 1 has twice as much entries as 2? As I see it output will have fewer entries than 2 because many entries in 2 have no match in 1. Is that correct assumption? Thanks.
- 11 years ago
-
Contest Holder - 11 years ago
it should not matter as we only want to know the entries that (partially) match.
- 11 years ago
-
Contest Holder - 11 years ago
Its true that list 1 has more entries, and that there are not so many matches. However we want to know the entries that do (partially) match. I suggest you use list 1 and find matches from list 2
- 11 years ago
-
IZABS
- 11 years ago
I almost done can u plz send me your email id so i send you. Thanks
- 11 years ago
-
Contest Holder - 11 years ago
please send it to support [at] lead boxer.com
- 11 years ago
-
SrinivasaBagade
- 11 years ago
Please clarify query language will do?
- 11 years ago
-
Contest Holder - 11 years ago
yes, sql is good.
- 11 years ago
-
CabbNetic
- 11 years ago
do you want the end file to be an ms excel file ? please reply so that i start working on it.
- 11 years ago
-
Contest Holder - 11 years ago
End file should be an script/application/code that does the matching.
the data in list-1 and list-2 are samples- 11 years ago
-
SrinivasaBagade
- 11 years ago
Will SQL of query sending data to .csv work
- 11 years ago
-
razorgeneral
- 11 years ago
Hi. I'm not clear on what the matching score is ? Can you explain ?
- 11 years ago
-
Contest Holder - 11 years ago
This can be your own matching score or something like Levenshtein distance
- 11 years ago
-
Jnkrgl07
- 11 years ago
Hi! I would like to know how is the matching score established.....
- 11 years ago
-
Contest Holder - 11 years ago
This can be your own matching score or something like Levenshtein distance
- 11 years ago
How to get started with contests
-
Post Your Contest Quick and easy
-
Get Tons of Entries From around the world
-
Award the best entry Download the files - Easy!
