I have 26 million urls listed one per line in a text file. I need all duplicates removed.
26 Million urls listed one per line, divided across 26 text files containing 1 million each. I need the duplicates removed so that I have 26 million unique urls. Then simply split the urls across 15 text files and return to me. I need this done TODAY!