[sunjester tuts] Combolist Management in C#: Part 1

[18]

Introduction

You can follow along by reading the first tutorial about loading/running C# on Linux here. If you have already read that one or know how to do it, please feel free to continue. You can read my other combolist management thread (in bash/shell script) here. All of my tutorials are written under Linux, and I will not pander to the Windows kiddies. I have been developing software since before .NET was around and normally don't even use an IDE, I either use SublimeText, vi, or nano. Nothing wrong with using a full IDE, especially if you're new.

Combo Lists
You can get a shit ton of combo lists from my old github repo (https://github.com/therealsunjester/combos) that has 17 million combos. You can use any of those to practice your skills and use the code presented here. Most of the combo lists people are using are huge. When you are dealing with a large amount of data, reading line by line into a string[] array is just stupid, it will slow your computer down and probably crash your software. The lists in my repo shouldn't have duplicates in them and they should all be a colon delimited format.
 
We can read a list (the list im using has 8,000 lines) into a generic list:

Code:

public void RemoveDupes(string fileName)
{
    var file = File.ReadAllLines(fileName);
    var lines = new List<string>(file);

In the code above we are reading a file named fileName into a generic string list named lines. Two libraries are needed for this: System.Collections.Generic and System.IO.
 
Removing Duplicates
When removing duplicates you can use Linq's Distinct method since we are using a generic list to hold the combos. It requires a simple addition to one of our lines of code.

Code:

var lines = new List<string>(file).Distinct();

That's it, you have removed all the duplicates from that generic list. So now that we have remove the duplicates we can write it back out to a new file named "no_dupes". So let's put it all together.
 

Code:

public void RemoveDupes(string fileName)
{
    var file = File.ReadAllLines(fileName);
    var lines = new List<string>(file).Distinct();
    TextWriter tw = new StreamWriter("no_dupes");

    foreach(string line in lines)
    {
            tw.WriteLine(line);
    }

    tw.Close();
}

Dealing with HUGE Combo Lists
When dealing with large combo lists the best thing to do is read a bit at a time. There is no reason to ever store that much in a single file but if you do (and you know who you are) we can use a temporary file and read a little bit at a time. This will keep your software from locking up and your computer free from dying. This is a 24mb flat file with 800k+ combos https://github.com/therealsunjester/comb...r/815K.txt. This is not something you should load into a string[] or even a generic list all at once.
 
So what do we do? Chunk it up into smaller sections.

Code:

public void ReadLargeCombo(string fileName, int start, int howMany)
{
    List<string> lines = new List<string>(File.ReadLines(fileName).Skip(start).Take(howMany));
    foreach(string line in lines)
    {
            Console.WriteLine(line);
    }
}

So to use the method we would supply the name of the file, which line to start with and then how many to return from the line we started at, for example:

Code:

combos.ReadLargeCombo("combolist1", 50, 100);

We are starting at line 50 in the file and returning 100 lines.
 
Conclusion
I am stopping here and going to provide you with the class I've written so far. I changed the ReadLargeCombo method to return the array of lines instead of printing them out on the spot (since that would be bad OOP practice). If you have suggestions for the next tutorials, let me know. If you have problems with your combo lists and would like to have a solution to the problem, let me know, I will add it to the next tutorial.

Code:

class ComboSun
{
        public ComboSun() {}

        public void RemoveDupes(string fileName)
        {
                var file = File.ReadAllLines(fileName);
                var lines = new List<string>(file).Distinct();

                TextWriter tw = new StreamWriter("no_dupes");

                foreach(string line in lines)
                {
                        tw.WriteLine(line);
                }

                tw.Close();
        }

        public List<string> ReadLargeCombo(string fileName, int start, int howMany)
        {
                List<string> lines = new List<string>(File.ReadLines(fileName).Skip(start).Take(howMany));
                return lines;
        }
}

[4]

when are you going to release the second one???!

---

this is cool