I have 26 million urls listed one per line in a text file. I need all duplicates removed. Text file is zipped in 670meg file so please allow time to download and upload.
26 Million urls listed one per line, divided across 26 text files containing 1 million each. I need the duplicates removed so that I have all unique urls. Then simply split the urls across 15 text files and return to me. I should have about 15-20 million uniques in the end. I need this done TODAY! File size is 670meg so allow for time to download and upload.