Archives

Categories

Web Site Validation

Over the last few days I’ve got this blog and my documents blog to conform to valid XHTML according to the W3C validation service [1].

One significant change that I made was to use lower-case for HTML tags. For about 15 years I’ve been using capitals for tags to make them stand out from content and my blogs are the latest in a long line of web sites with that. Naturally I wasn’t going to correct 900 posts manually so I ran a series of SQL commands such as the following on my database server (where X is the WordPress table prefix):

update X_wp_posts set post_content = replace(post_content,'<PRE>','<pre>');

But make sure you have a good backup of your database before running SQL search and replace commands on your blog data.

After running such commands about 90% of my blog posts conformed, so I only needed to edit about 90 posts to correct things. This process gave some real benefits. One issue is that an apostrophe in a URL must be quoted, otherwise some browsers will link to the desired URL and some will link to a truncated URL. Fixing a couple of variations of this problem resulted in some broken links being fixed. Another issue is that you can’t have paragraphs (<p> tags) within list items, fixing this made some of my posts align correctly – it was a tricky fix, in some cases I had to use <br/> to break up text in a list item and sometimes I replaced lists with different sections delimited by <h3> headings (which apparently is rumored to give better SEO).

It would make a really nice WordPress feature to be able to do W3C validation as part of the publishing process, ideally an attempt to publish or schedule a post would result in a message saying “saved as a draft because it’s not valid XHTML” if the checks failed. The source to the W3C validation software is significantly larger than WordPress [2], but it seems to me that there are two main types of WordPress installations, small ones for personal use (which tend to be on fairly idle servers) and big ones that have so much traffic that the resource usage of validation would be nothing compared to the ongoing load.

As there seems to be no way of validating my posts before publication my best option is the W3C button I now have on my blog. This allows me to validate the page at a click so while I can’t entirely avoid the risk of publishing a post with invalid XHTML I can at least fix it rapidly enough that hardly anyone will notice.

It also seems like a useful feature to have aggregators like Venus [3] check for valid HTML and not display posts unless they are valid. It’s not a feature that could be enabled immediately (I’m sure that if you click on this link to the W3C validation service [1] from a Planet feed you will see lots of errors and warnings), but once bloggers have time to fix their installation it would allow preventing some of the common annoyances of Planet installations. It’s not uncommon on popular Planets to have unmatched tags in a post which results in significant amounts of the content being bold, underlined, in italics, or for the greatest annoyance struck-out. I know that this may be a controversial suggestion, but please consider why you are blogging – if you are blogging for the benefit of your readers (which seems to be the case for everyone other than sploggers) then it seems that the readers will benefit more by not having a broken post syndicated than they would benefit from having it syndicated and thus messing up the display of many following posts.

The next thing on my todo list in this regard is to do some tests of accessibility. The work that I have done to pass the XHTML validation tests has helped to some degree – if nothing else the images now all have alt= descriptions, but I expect that it will be a lot of work. The WordPress Codex has a page about accessibility, I haven’t read all of it yet [4].

Does anyone have any recommendations for free automated systems that check web sites for accessibility? What would be ideal is a service that allows different levels of warnings, so instead of trying to fix all problems at once I could start by quickly fixing the most serious problems on the most popular posts and finish the job at some later date.

Security and Hiring

The main sources of information used when hiring someone are their CV, the interview, and references.

CV

The CV is written by the applicant or sometimes for the applicant. Naturally it says only good things, if a CV notes no skill in a particular area then it may be used to exclude an employee from consideration. But the trend is towards including a reference to anything that you touch, so someone who lists DBA experience may merely have done a couple of CREATE TABLE operations.

Interview

The interview is a good test of people skills but is often of little value in assessing technical skills. The interviewer asks questions such as “do you know technology X” and the applicant says “I know that really well“. If the company is hiring another person with similar skills to current employees then they can have their current employees sit in on the interview and ask difficult technical questions, but for unknown reasons managers often don’t take that option and get no advice from their technical people. Also if the company is hiring someone with specialised skills (EG they are about to implement a new application and want to hire their first employee to work on it) then it may be impossible for them to assess the technical merit of answers. Probably the best use of the interview is to match answers with the CV, if the applicant doesn’t appear to know the contents of their own CV then they should be rejected.

The biggest problem with interviews is when the questions are all of the form “do you know X“. Someone who really knows it will say “yes” as will someone who doesn’t know enough to realise the limits of their knowledge – and such ignorant people vastly outnumber the skillful people. The real problem is that the people who are moderately skillful will lose out. If someone asks me about my MySQL skills I will tell them that I’m not really good at it. Sure I’ve run replicated servers with tens of thousands of users running 24*7, but that doesn’t mean I’m really good at it – probably most people who will claim to be great at MySQL without qualification would have less experience than me.

References

Reference checks rely on an unknown person saying good things about the applicant. For starters there is the issue of the number of references which may not be representative of their employment history – EG the applicant could use as a reference the one manager who didn’t sack them.

The next issue is that there is little incentive for the referee to be honest, most people are aware of instances where someone once worked for a friend and can rely on good references for the rest of their career. If a reference is inaccurate then there is no realistic opportunity for redress.

Finally every reference check that I am aware of (checks where I have been the referee or the applicant) has involved the applicant giving the phone number of the referee to the hiring manager! The phone could be owned by a friend or relative of the applicant, so logically a good reference that is based on trusting the applicant to supply the phone number only proves that the applicant is either good or really bad. To make a reference check prove something the recruiter would at a minimum have to phone the number listed in the white-pages for the corporation that used to employ the applicant, asks to speak to the manager of the relevant department, and then gets a reference. Calling a mobile phone number that is supplied by the applicant (which seems to be the standard practice) is essentially trusting the applicant – and trust is the root cause of most security problems!

Really most of this ends up as trusting the applicant to provide honest evidence that they are trustworthy and believing that the applicant’s technical knowledge is good enough to be correct when they say that their technical knowledge is good. It can fail spectacularly when someone isn’t trustworthy enough to provide honest evidence of their integrity or when someone doesn’t have the skills needed to know that their skills are lacking.

As an aside, even if the reference is given accurately and in good faith it may still be misinterpreted. The fact that telephone references are exclusively relied on exacerbates this problem. Ideally references would be in writing with some way of proving their authenticity (maybe using phone verification of the accuracy of the written document).

Solutions

So how can we solve this? Some people believe that career based social networking software will solve the problems, but as usual I think that software doesn’t magically solve human problems. The first challenge when trying to use social networking to solve the problem is to find someone on your friends list who has relevant knowledge, this may be viable in a small industry (EG when someone from bank A applies for work as bank B in the same city). The next issue is that of false “friends“. I’m sure that I’m not the only person who has been pressured to add people as friends on social networking sites, the non-computer social interactions really don’t prepare people for saying “no you are not my friend” (apart from high-school I guess). With professional social networking sites there are further issues, if you are working on a client site and a manager demands that they be listed as one of your friends then what are you going to do?

So it seems to me that the social networking sites are at best a helper for the gossip network. If you think that a friend of a friend from a social networking site might be able to help you then you first ask your friend if the person in question is really a friend, and if so are they one of the shifty pseudo-friends you only hang out with because their company pays good money. But the problem with the gossip network is that it’s mostly secret and is therefore subject to settling vendettas, I’ve heard of senior managers going out of their way to spread false stories about former employees to settle scores.

The best solution I can think of is for someone who has a reputation to publicly stake it on the accuracy of their references. If I’m going to give a reference then I would be happy to do so via a GPG signed email or a blog post. This doesn’t mean that my references will always be correct, but it would show that I try to give good references.

Michael Atkinson Lies

The South Australian government wanted to force bloggers to disclose their name and post-code when commenting on an election [1]. According to Adelaide Now this included posts on Twitter and Facebook [2].

As expected there was a strong public reaction to this and Michael Atkinson (state Attorney General) stated that the law was not going to be enforced and that it would be retrospectively repealed after the election [3]. Which might have been the end of the issue, but he also said “All MPs and all parties voted for Electoral law. Hope Libs, Greens, Family First, Independents etc will join us to support repeal” which seems to be a clear claim that the Greens supported that legislation.

The Greens are pretty good about freedom of speech issues so I immediately enquired as to what was going on. The following is from a Greens media brief issued yesterday by the office of Mark Parnell MLC (the Greens MLC in SA) which was issued before the new law was retracted:

Attorney General Michael Atkinson must urgently clarify the scope of new electoral laws covering public comment on the internet, in the wake of concerns that the impact will be much wider than expected.
“With only weeks to go before the election kicks off, there is an urgent need for the Attorney General to explain exactly what his laws are
intended to capture,” said Greens MLC Mark Parnell.
“When this was debated in Parliament last year, we were told that the law change would only affect ‘electronic versions of a journal’. Now there is
concern that the laws could extend further, restricting public commentary on media sites like AdelaideNow and ABC Online,” he said.
When the Bill was debated in the Lower House, Michael Atkinson originally wanted to include ALL material on the web. However, the Government
backed down on this by the time it got to the Upper House, with Minister Holloway saying:
“The intention is to limit the coverage of section 116 as it applies to the internet to electronic versions of a journal rather than any electronic publication on the internet.”
A ‘journal’ was narrowly defined as ‘a newspaper, magazine or other periodical’.

Now I don’t have a great objection to a law that demands that journalists identify themselves when commenting on an election, and I think that most people would not care about that. It seems that Michael Atkinson is repeatedly changing his claims to try and match popular sentiment.

He has got form for this sort of thing, his past “achievements” include censoring the censorship debate about an R18+ rating for computer games [4].

3G Broadband for Home Use

I have just installed an old Three mobile phone with 3G broadband for my parents home network access for the reasons described in my cheap net access in Australia post [1].

The first problem I had was that the pre-paid Three SIM just wouldn’t work at all. I ended up phoning the Three support line and had a guy guess at which version of Windows I was running, after guessing every version of Windows from the last 10 years and Mac OS/X he finally asked what OS I use and then told me that Linux isn’t supported. I said “I HAVE TWO SIMS FROM THREE, ONE WORKS AND THE OTHER DOESN’T, IT’S ON THE SAME PC WITH THE SAME 3G ACCESS DEVICE, THE PROBLEM IS WITH THE SIM OR THE SERVER NOT MY OS“. When the support guy discovered that one sim was pre-paid he said that there is a configuration difference, instead of an APN of “3netaccess” for post-paid (contract) you have to use “3services” for pre-paid.

There are a bunch of web pages describing how to get Three 3G broadband working on Linux in Australia, some say to use 3netaccess and some say 3services. None of the pages I read stated correctly that 3netaccess is for when you are on a contract and 3services is for pre-paid. I’ve submitted a suggestion for the Ross Barkman’s GPRS Info Page (which seems to be the best reference for such things) [2].

After getting the pre-paid 3G SIM working for net access from the Huawei E1553 USB 3G modem I was unable to get it working from my LG U890 mobile phone. I never figured out how to solve this problem, I left my parents with the SIM that is connected to my $15 per month contract plan for 3G net access and am now using the pre-paid SIM for my own use. Of course this means that as I’m using a SIM registered to my mother and she’s using one registered to me I’ll surely have some problems getting the support center to help me with problems in future.

I found that the 3G net access got better reception when the phone was higher than the computer, so I used a USB extension cable to allow it to be placed on a shelf above the computer. The extension cable also allows it to be easily unplugged and plugged in again – I’ve already seen one situation where Linux got confused about the state of the USB device and replugging it was necessary to solve the problem. I was using Debian/Lenny.

Here is my chatscript for connecting to Three with my 3G modem on a pre-paid SIM – which also allows roaming to Telstra (I haven’t tested whether pre-paid allows roaming, I’ve only tested Telstra roaming with a contract SIM):

ABORT 'BUSY'
ABORT 'NO CARRIER'
ABORT 'ERROR'
'' AT
OK ATQ0V1E1S0=0&C1&D2+FCLASS=0
OK 'AT+COPS=0,0,"3TELSTRA",2'
OK AT+CGATT=1
#OK AT+CGDCONT=1,"IP","3netaccess"
OK AT+CGDCONT=1,"IP","3services"
OK ATDT*99**3#

Here is the ppp configuration for connecting via the USB 3G modem. For use as a permanent connection you want to also include persist and “maxfail 0“:
/dev/ttyUSB0
230400
noauth
defaultroute
logfile /var/log/ppp.log
connect "/usr/sbin/chat -v -f /etc/chatscripts/three"

For connecting with an LG U890 mobile phone you need to use “ATDT*99***1#” as the dial command and the device is /dev/ttyACM0 .

Precision vs Accuracy in Identifying People

Andrew Dowdell and Michael McGuire have an interesting article in the Adelaide Now about censorship in the South Australian election [1]. The South Australian government wants to force everyone who comments on the upcoming SA election to provide their name and postcode. Attorney-General Michael Atkinson said the law was “all about honesty“. However a law that forces someone to not comment as “Anonymous” and instead forces them to use a name that sounds like something that might appear on a birth certificate and a postcode is not going to increase honesty at all.

I think it’s much better to honestly say “I’m not telling you my name” than to lie and claim to be revealing your name. When the government forces people to give precise but totally inaccurate information it seems that it’s going to be bad for everyone. It’s even bad for the people who want to be identified, if most people who comment on a blog post are anonymous then I’d like to be distinguished as the person who uses their real name!

In the vast majority of cases the effort of determining who is using their real name will not be worth the effort. If someone comments on my blog under the name “John Howard from Bennelong” I will be very suspicious that they are using a fake name. But probably the vast majority of the English speaking population of the world wouldn’t immediately identify such a name as fake. If someone wants to comment on my blog and they don’t have a published phone number then I won’t have any good way of identifying their address. If their name and postcode match an entry in a phone book then they still might be faking it – they could take a random name from a phone book. If I was to demand that people who enter blog comments provide their phone numbers then I would have to pay the expense of phoning them as well as dealing with cases of people who are away from home, should I delay a blog comment for a month until the author returns from their European vacation?

I have had a moderate amount of experience in writing letters to the editors of newspapers, and I have only once had an editor phone me to verify my details. In all other cases I guess I could have fooled them if I wished.

I believe that there are already precedents regarding libel, if I approve a blog comment (or fail to unapprove it in a reasonable amount of time if it meats the automatic approval criteria) then I could be sued for libel if the contents of the comment are deemed to be suitably damaging. So if I was to try to get the real names of people who make comments on my blog then it wouldn’t make it any easier for an idiot who wants to sue – also anyone who wants to sue regarding an Australian political issue will probably find me a better target than most people who comment on my blog (a significant portion of whom are from the US and have much greater legal protections of their freedom of speech).

Extending such a law to US based services such as Twitter is just silly. The stockholders and employees of a US based corporation can freely laugh at Australian censorship laws. Also it’s pretty stupid to have a global scope on such laws, as a Victorian why should I care about the South Australian state laws? Implementing laws that can be easily broken inadvertently and can never be enforced against anyone who cares is just pitiful and will result in the MPs who vote for such laws being the object of derision.

Vote for the small parties and independent candidates. Both Labour and Liberal want to censor us, put them in the second last two spots on your vote card!

Now this law doesn’t take affect until the writs for the March 20 election are issued. I encourage Australian bloggers to write bad things about the Liberal and Labour parties after the writs are released which are not libelous and which don’t include your postcode. If they try to apply the new law then your blog post gets wide attention. If you plan to be a professional blogger then you could consider the $5,000 fine to be an advertising fee. If you don’t want to be known then you can use a US based blogging service.

I’m going to continue to write political blog posts whenever I feel like it and I won’t be telling anyone where I live. I will rely on the Streisand effect to save me.

Update:
The politicians in SA have surrendered, the law in question won’t be enforced and will supposedly be repealed after the election [2]. It will be interesting to see whether they really do repeal that law.

Choosing an Australian Mobile Telco for use with Android

Since playing with the IBM Seer augmented reality software [1] I’ve been lusting after a new mobile phone which can do such things. While the implementation of Seer that I tried was not of great practical use to me (not being a tennis fan I was only there to learn about computers) it was a demonstration of an exciting concept. It will surely be implemented by IBM in other venues that are of more immediate interest, and we can probably expect other vendors to write similar systems to compete with IBM.

So the question is how to get a phone that will run such things well. The answer is probably not to rely on a contract plan for this, currently Vodaphone [2] is currently the only Australian telco that sells a phone that can run Seer, it is offering a HTC Magic (which was released in April 2009) on a $29 per month plan. A phone that is 9 months old isn’t necessarily a bad thing, but the has been out for more than 6 months and has some significant benefits (such as a 5MP camera).

  • My current provider is Three [3] and their cheapest plan is $29 per month (with or without a phone) which allows 200 minutes ($160) of free calls to other Three phones every month as well as up to $150 of other calls per month and 1GB of data. Calls cost 40c per 30 seconds plus 35c connection fee. Currently I’m on a plan that gives me the same thing for the same price without data transfer but which includes a “free” phone. So it seems that the 1GB of data per month has an equal cost to a mobile handset (such as an LG Viewty).
  • Virgin [6] has a $25 per month plan that gives $60 worth of calls and 300MB of Internet data with unlimited talk and text between members with the added bonus of unused talk and text credit being rolled over to the next month. The cost for calls is 90c per minute plus 40c connection fee, video calls are the same cost as voice calls!
  • Vodaphone [2] has a $20 per month phone plan that allows up to $150 of calls per month with the option of either free calls to a single specified Vodaphone number or free calls in the evenings and weekends. They have a $4.95 special offer for 200MB of Internet data per month. Calls cost 44c per 30 seconds plus 35c connection fee.
  • Telstra [4] has a $20 per month plan that only includes $20 worth of calls and which has call fees of 47c per 30 seconds plus 27c connection fee. They clearly don’t compete on price, I think that there is no reason for using Telstra unless you live in some of the rural regions where they are the only provider to offer good service.
  • Savvytel [7] charges $3.07 for GPRS Internet data so they can’t be considered for an Internet enabled phone. But they do seem very economical for basic phone service.
  • Optus [5] has a hopelessly broken web site that wouldn’t give me any information on mobile phone pricing. My previous experience with Optus Internet makes me unlikely to do business with them again anyway.

So it seems that Three (my current provider) is probably the best option at this time. Virgin would save my $4 per month, but would only give me 300M of Internet data per month, and the Virgin limit of $60 per month of calls might not be enough for me. Vodaphone offers a deal for $25 that only includes 200MB of data, that might be enough for just phone use, but wouldn’t be enough for tethering for laptop net access.

I wonder how well tethering works on an Android phone, can you make a phone call while transmitting data from a tethered laptop? I find that with my Viewty when I receive an SMS or phone call it stops the net access. That makes a tethered Viewty impractical for some support tasks as it’s fairly common that I need to talk to someone while logging in to their server – I’m sure that most people who use mobile Internet services regularly need to phone someone while using them.

My current Three bill is $29 per month for the phone plan and $15 per month for Internet access. If I’m going to buy phones outright instead of getting them with the plan then I want to reduce the overall amount of money I spend on phone plans and using tethering instead of a 3G USB dongle would allow this. I think that I can get something that comes close to my ideal mobile phone [8] (apart from being able to connect a keyboard, mouse, and monitor) if I import it from overseas.

We really need more competition in the Australian mobile phone market. We have only two phone companies offering Android phones, Three is sold out of the obsolete model that they offer while Vodaphone has stock of an obsolete model.

My Ideal Mobile Phone

Based on my experience testing the IBM Seer software on an Android phone [1] I have been considering what type of mobile phone to get when my current contract expires. Here are the features above what is common in current smart phones that I think most people will sorely miss if they don’t have them for the 2011-2012 period:

  1. Camera that takes reasonable quality pictures at a 5MP resolution.
  2. High resolution screen (VGA or better).
  3. GPS (for navigation and augmented reality.
  4. Digital compass for augmented reality.
  5. An open market for applications which allows free software to be installed – such as OpenSSH.

The first two items shouldn’t be a problem, there has been a constant trend towards better cameras and higher resolution screens in phones. The difficult ones are GPS and a Digital compass which require phone software to use them. I get the impression that Android and iPhone are going to share the market for fully functional smart phones (because they have the market of applications). So I predict that by 2012 the phone market will have iPhone and Android fully functional smart phones as well as budget phones that don’t support running applications (and will probably lack a compass and GPS).

Here are the features that while not essential, will greatly increase the experience of using a phone for serious users:

  1. At least 2G of storage built in – installing a 2G micro-SD card is not adequate.
  2. A screen that can be easily read during the day – maybe Pixel Qi.
  3. The ability to give a good quality of sound for playing video and audio recordings with a regular headphone jack (so I can use my Bose headset).

For my use a hardware keyboard (such as is used in the Motorolla A855 “Droid”) is essential. I want to have a pocket sized ssh client for emergencies, and I want to be able to type notes reasonably quickly.

I wonder what portion of the smart-phone user base actually needs a keyboard. I’ve seen many people who use a smart-phone as just a regular phone that can exchange photos. Even among people who are moderately serious about smart-phone use there are probably many who only want to take high resolution photos and tag them with GPS data. Currently there are no Android phones on sale in Australia that have a hardware keyboard, I’m worried that this may be an ongoing trend which will result in people with my requirements being forced to either pay significantly more or compromise on features due to the market meeting the needs of average people.

Finally I would like to have a smart-phone that has a regular USB port for plugging in devices (which would of course require an adapter as the size of a phone doesn’t permit a regular USB port). That would permit copying files from USB flash devices, driving a digital SLR camera, and printing photos directly to a USB printer. It would also allow connecting a USB video device, keyboard, and mouse to make a mobile phone work as a desktop workstation. Current smart phones have a lot more compute power than the desktop machines I was using in 1998, so there’s no reason that one couldn’t be used as a workstation with the appropriate peripherals.

The Australian Open and Android Phones (Seer)

On Monday the 25th of January 2010 I visited the Australian Open [1] – it’s one of the world’s greatest tennis championships and it’s on in Melbourne right now. IBM sponsored my visit to show me the computer technology that they use to run the event and display the results to the world via their web site and to various media outlets.

picture of IBM Seer software running on HTC Hero

The first thing that they showed me was the IBM Seer software on the HTC Hero phone (which runs the Google Android OS). Seer can be freely downloaded from the Android store. The most noteworthy feature is that it uses the camera in the Android phone to display a picture of whatever you are looking at with points of interest superimposed (such as the above picture where I asked for locations of events and toilets). But it also displays a map view and has some other features I didn’t get a chance to test such as viewing twitter data relevant to the event. We really need this augmented reality feature enabled with tourist data for major cities. I’m sure that there’s lots of interesting things I haven’t seen in my own home city, if I could just pull out a phone and see a map of what’s around me whenever I’m bored I could see some of them. I think that this has the potential to change the way we use phones, in theory this was available as soon as Google Maps was released, but Seer seems to be the start of a whole new range of developments. One of the uses of this will be for identifying the background in tourist photos, no more of the “me in front of old building” descriptions.

Here is a Youtube Video of the Seer software in action [BMLgHGV4zWM].

The tour guide explained that to get the software to work on an iPhone would require the 3GS for navigation as the earlier iPhones don’t have a compass. The Seer software was initially developed for The Championships, Wimbledon 2009 which happened at about the same time as the iPhone 3GS release. I expect that there will be enough iPhone 3GS units sold before Wimbledon 2010 to give IBM a good incentive to port Seer to the iPhone.

Three (my phone company at the moment) has just sold out of the HTC Magic which has a digital compass. They are selling the HTC Touch Pro and HTC Touch Diamond that appear to lack a digital compass. Vodaphone is offering a HTC Magic free on the $29 contract right now. The other Australian mobile phone companies don’t seem to offer any Android phones. So it seems that the only option right now if I wanted to purchase a phone in Australia that can run Seer is the Vodaphone HTC Magic, and that’s a phone that was released almost a year ago (a long time with recent progress in phone development – it’s the modem before the HTC Hero I tested) and which has only a 3.2MP camera (the LG U990 Viewty I used to take the picture for this post has a 5MP camera and is older than that). So I expect that there aren’t many people using Seer in Australia.

If you happen to be in Melbourne and have an Android phone with a digital compass then you may want to visit the Australian open to try the Seer software. It should work equally well from outside the security fence…

I’ll write about the other things I saw over the next few days.

Costs and Benefits of Search Engines

Chris Smart writes about the latest money making schemes for OS distributors, Canonical is getting paid by Yahoo to make them stop using Google as the default Firefox search engine [1]. I think this is OK, the user can easily change it back if desired and it allows them to pay the salaries of more employees – who contribute code back to upstream projects.

MSN uses 455M, Google uses 189M, and Yahoo uses 109MGoogle referrs 8250 hits, but Bing only refers 280

Above are sections of my Webalizer output related to my blog which show the data transfer use of search.msn.com (Bing presumably) which is 50% greater than that of Google and Yahoo combined. Why does MSN need to do 455MB of transfer so far this month to scan my blog when Google gets the job done with 189M and Yahoo only takes 109M? Also judging by the referrals Bing is only 3% as much use to me as Google.

MSN uses 525M, Yahoo uses 132M, and Google only uses 35M

Above is a sample of the Webalizer output from www.coker.com.au, MSN is using 525MB of data to scan the site which contains about 1.2G of static files that change very rarely. A Russian malware site seems to be downloading it three times a month, and Google only takes 35MB of data transfer to scan the site (which is probably still excessive).

If Bing was a quality search engine that returned appropriate results then this could be forgiven. But however it is a very poor search engine that returns bad results. For example if you query Google or Yahoo for “bonnie++” you will get an entire page of search results concerning my Bonnie++ benchmark, and those results are ordered in a sensible way. If you ask Bing then the first four results concern “Bonnie” (three women and a plant) and most of the first page don’t concern my benchmark.

Some time ago I had blocked MSN from scanning a server that I ran. The server in question had all the web servers for my domain plus quite a few other small domains. The total MSN data transfer was 3G per month which was almost half the data allowance for the server in question (data plans in Australia suck – that’s why my web servers are hosted in Germany now), so it was a question of whether to allow normal operation of the business or MSN searches. With Microsoft not running a popular search engine (then or now) it was an easy decision.

I think that anyone who accepts money from Microsoft/Bing is doing their users a mis-service. Bing is simply an inferior search engine, it gets bad results and imposes excessive costs on service providers. Yahoo however seems to be a reasonable service, not as good as Google for the web hosters but not too bad.

I wonder what would happen if Yahoo offered some sponsorship money to the Debian project in exchange for being the default search engine. I’m sure it would be dramatic.

Discovering OS Bugs and Using Snapshots

I’m running Debian/Unstable on an EeePC 701, I’ve got an SD card for /home etc but the root filesystem is on the internal 4G flash storage which doesn’t have much spare space (I’ve got a full software development environment, GCC, debuggers, etc as well as running KDE4). On some of my systems I’ve started the practice of having two root filesystem installs, modern disks are big enough that usually it’s difficult to use all the space, and even if you do use most of the space the use of a second root filesystem only takes a fraction of a percent of the available space.

Today I discovered a problem with my EeePC, I had upgraded to the latest Unstable packages a few days ago and now when I run X programs the screen flickers really badly every time it’s updated. Pressing a key in a terminal window makes the screen shake, watching a video with mplayer makes it shake constantly to such a degree that it’s not usable. If that problem occurred on a system with a second root filesystem I could upgrade the other a few packages at a time to try and discover the root cause. But without the space for a second root filesystem this isn’t an option.

I hope that Btrfs [1] becomes ready for serious use soon, it seems that the btrfs snapshot facility might make it possible for me to preserve the old version in a bootable form before upgrading my EeePC (although even then disk space would be tight).

So I guess I now need to test different versions of the X related packages in a chroot environment to track this bug down. Sigh.