Algos are secret to prevent abuse

First off, abuse happens anyway, since while the actual algorithm is secret, SEO experts can quickly figure out what causes their rank to go up and down.

Secondly, and more importantly, is the difference between security through obscurity and security through, y'know, actually building a secure system.

Both have their merits, but the problem with obscurity is that as soon as a weakness is discovered, it gets posted online, and only then do a few priviledged programmers get to work on fixing the problem, while an army of bored hackers exploit it.

Obviously building a secure search engine that works the way that Wikia is intended to is a huge challenge. But the advantage that open source has is that it exploits a massive talent pool which results in more creative solutions to problems.

I don't know if the project will succeed or not. It all depends on how many programmers and how many well-meaning, contributing users it can attract.

remember those search sites from back when, where humans organized the sites for you into categories, etc.? well, nowadays wikipedia has become a well organized front end for web pages, and google indexes wikipedia.

Website managers and online publishers expend enormous effort to improve their ranking in search engines - it's called search engine optimization, or SEO.

But when we enter a query into a search engine, we aren't looking for the website that is best at SEO. We want websites with the best content. If the algos were open source, it would be too easy to be really really good (and sneaky) about SEO. So you wouldn't get the best websites in your results, you'd get the sneakiest.

There used to be a very highly transparent way for search engines to rank pages against keywords - the keyword meta data in the HTML document. But those meta data fields in HTML documents became so abused and convoluted that today they are almost entirely ignored by search engines.

Let's keep it in perspective.

Google: Gives us links to content with lots of inbound links

YouTube: Gives us 2x4 inch silly videos of home movies

Wikipedia: Gives us access to the all the world's information, updated by experts in every field.

Hmm. Wikipedia may not be perfect, but it is still the most brilliant invention of the current Internet generation.

The reality is current search engines are expert in feeding Attention Deficit Disorder, with all the sites frantically trying to game the system or all the marketers trying to buy attention. That works, to a point. If I want the latest scoop on Britney Spears, I'll use Google and scan left. If I want a deal on car tires, I'll use Google and scan right. But if I want deep, meaningful, well-written, knowledgeable stories or analysis or video, Google and MSN/Live and Ask.com rarely get me there without digging. Perhaps humans, given enough time to build a wise crowd, can.

Every time Google adjusts its ranking system, hordes of increasingly well-heeled search engine spammers tune up their link farms and spam sites to push their vapid results toward the top. Fighting this is a major effort for Google and its competitors. What Wales envisions contains the seeds its immediate collapse in this environment.

Search engines would rank nothing but porn and celebrity bashing sites.

I know that Farhad's whole shtick is playing the uber-tech geek, and waxing embarrassingly overwrought about the latest fad. But Wikipedia

is not

"the world's best reference source." It's not close. It's not close! And, as anyone who has contributed to the project can tell you, it won't ever get there, because the open-ended nature of the project means that it gets worse as well as better. (Ask a dedicated Wikipedia editor, and if they aren't engaging in propaganda, they'll tell you that articles being fixed and then reverted back to inferior form happens all the time.

And as searchers, do we really want or need that transparency? Ten years ago I could look under the hood of my car and fiddle with my engine when I wanted to modify something. Today, just about every system in my car is computerized, completely inaccessible to my tinkering. But given the virtually maintenance-free operation of my car I'm perfectly happy with that change and don't long for the lost days of "engine transparency" at all.

No, as I searcher I don't need that transparency, but I certainly would like others to be able to figure out what's causing the results to come up the way they are.

There's a vast difference between a car (which you can tell, instantly, whether it successfully got you from point A to point B) and a web search (where you'll never really know what results you aren't seeing). And we live in a world where political and commercial entities have a interest in limiting the information we can see.

I currently trust Google to rank and filter results out of utility, not censorship. But that doesn't mean there's no value in an open community-driven search engine. I may not trust Google forever. And with Wikia Search, at least you'd know what the filters were.

Google is a thousand data centers, bleeding edge mathematics, brand new software technology. Does he think that Wikinauts are going to replace all that with volunteer sweat?