Rogue deck is a term that can mean two similar but different types of Magic deck. It is most commonly used to refer to a deck which does not seem to follow the trends of a particular metagame. Rogue decks are often attributed to Johnny players, and are generally not believed to be able to compete with the more popular decks in the metagame. Such decks are usually classified as being below Tier 1 decks in any given metagame, but are just relevant or common enough to warrant discussion. This does not mean that such decks are bad as such, and indeed many of them are designed specifically to beat a dominant deck but this almost always comes at the cost of being less effective against the rest of field. In this case players piloting such decks at tournaments will often be gambling that one certain deck that they have a very favorable match-up against will make up enough of the field for them to play almost exclusively against that deck and thus allow them to do well.
Alternatively, Rogue Deck can mean a deck that is developed secretly in the run up to a tournament and so arrives on the day completely unknown to the opponents giving the pilot the element of surprise. In this case, such decks are designed to beat an entire metagame normally through the use of cards or strategies which are not currently being used in the high level metagame. Decks of this type are often very powerful, but also fragile and only able to be consistently good before the metagame adjusts at which point they often become unplayable. This however is not a downside for the deck builder, because the aim is not for longevity or to make a new tier 1 deck, it is to win a specific tournament with a specific metagame and nothing else, with the player switching decks immediately after. However, when these decks are less disruptable and gain some longevity they frequently become some of the most powerful decks in MtGs history, warping the metagame around themselves. This kind of rogue deck, if successful, will normally be referred to as a format breaker, sweeping every other deck before it during the period when no deck has the tools to beat it and then even when the tools are ubiquitous continue to be of tier 1 power.
The building of Rogue decks, but sometimes simply decks in a not very well defined format e.g. after a rotation is referred to as brewing. Building a successful rogue deck is highly coveted and is generally considered to be the highest achievement for any deck builder. The challenge is in creating a deck that no-one else has considered, and then adequately testing and tweaking it to make it tournament ready without any input from the community and ensuring that the list isn't seen by anyone, no mean feat considering that most testing has to be done online where results and deck lists are available for all to see. Many of the most famous decks in history began as completely unknown rogue decks that destroyed a PT or GP and then proliferated into the metagame later on.
Examples of tournament-specific rogue decks (those which were successful then dropped off when players were prepared) include Seismic Swans from GP Barcelona in 2009, and Dragonstorm from Worlds in 2008.
Examples of rogue decks that became format staples include Dredge from GP Charlotte in 2006.