A way to spend Christmas Week

The time between Christmas and New Years is the slowest time of the year. Large portion of my team is on vacation and the customer activity is reduced to emergencies and routine maintenance.   For me it is an opportunity to spend more time with family, relax and… “geek out”.

As CEO large portion of my days consists with times of running the company taking all kind of high level decision managing projects and people, while I believe to be good at what I do I need to make sure to keep strong hands on skills and this slower period provides a lot of good opportunities for limited distractions while doing some hands on stuff.

Some previous years I would play with some PHP programming skills or setting up MySQL Zoo with MySQL Sandbox, while this year I was for real treat.

Few months ago a friend of mine send me bunch of old servers he was decommissioning from the data center. I got bunch of Dell PowerEdge 2850 and some smaller similarly aged beasts.   I have and old rack in my basement so I was setting them up as a server farm to later install Percona XtraDB Cluster and run some evil experiments like unplugging network cables which are just not as fun to do in the Cloud.

The old hardware is really great if your purpose it training troubleshooting skills as chances are much higher it might develop some weird problems and so you can twist your brain about how exactly you can diagnose and troubleshoot them.   You tend to forget how weird hardware can be if all you deal with is hardware managed by someone else in remote data center and the cloud.

Replacing the CPUs and testing memory is not considered high value skills. Typically college kids with little training can do them,  yet I think it is very valuable to go through these activities every so often to maintain good knowledge about details which can be very valuable troubleshooting complex problems.

For example over last couple of days I learned what database IO stalls could be pointing to bad memory in the RAID controller (even though it does not report any problem).   Another puzzle I’m still to resolve is why I have 2 servers installed absolutely same way but sysbench is showing very different IO performance numbers on their RAID controllers.

Bizarre ways of american elections

On the big election day in American I often end up thinking how bizarre American president election is, which in my opinion does not make the best leaders possible being elected as well as causes all kind of other issues.  Here are my two top concerns.

Electoral College based election   Might be it served its purpose 200 years ago but I think with modern national wide reach of media it causes more hard than good. First it it makes it unfair to american citizens as some votes are more valuable than others  – if you’re Republican in California or Democrat in Texas your vote makes essentially no difference compared to the votes in so called “swing states”.  Second with election focused mainly around the few swing states it is unfair to the states as presidents are trying to court the voters of swings states much more of states voting traditionally which essentially promotes the special interest politics on state level.  Popular  Vote based election would be my solution even though I recognize it would require changes to constitution and there is unlikely enough will to do that.

Two candidates based election    So there is candidate from Democrats,  Republicans and no one else essentially matters.   To have the chance to win majority of politicians need to align themselves with one of the two parties and try to win nomination, which frankly propels people who know how to work the system rather than strongest leaders to the top. Because of choice of 2 candidates people are often tempted to vote for lesser of 2 evils rather than supporting candidate they might really like which makes it hard for those candidates to get any visibility.   This even can be seen on primary elections when it quickly becomes about 2 top contenders of each party as presented to us by mass media. As result all election is about picking someone from established elite with little new blood and new ideas entering the politics.

The solution I see is direct president election as exist in many countries with president candidacy essentially unrelated to the party, though party may chose to endorse the candidate of its liking.  Some number of signatures might be needed for serious nomination so one can avoid candidates nominating themselves as a joke.   When 2nd stage of election can be ran as needed if no one wins 50% in the first round.

Such process I believe allow for much better expression of people’s opinion as in the first round people will vote for candidate they really like rather than lesser of 2 evils.  In addition to whomever wins it allows for better tracking for popularity of ideas which candidates represent.  For example  if we would have such elections we might have seen Ron Paul capturing say 5% or 15% of the vote so we would know how much popularity his ideas have – currently we have no clue as it was all hidden in the primary nomination game which does not even represent the full population.   Showing group of people their ideas are getting traction allows them to better organize and get resources for next elections and allowing new ideas to be represented a lot better in the politics.


Keep it green – do not upgrade your iPhone

We’re back again in iPhone mania season with so many people dreaming upgrading to yet another new gadget.   And there are really so many reasons to do that.

It is amazing how Apple and other phone manufacturers and carriers in US play together to encourage waste.   Most of people use contracts with their wireless carriers which subsidize phone purchase but which make monthly charges, especially for Data very expensive.

Rumor is the Apple getting $600 per device from carriers which is sold for $199 which means $400 per device is subsidized.  Divide it by 24 months (2 years) and you get some 17$ a month of your subscription fee is used to subsidize the phone.

I’m using Iphone 3GS for about 3 years now.  I must praise Apple for quality – it is scratched and cracked but it still works, holds battery life and can run all the applications I really need. It also works as a phone… which well it is.

Yet I do not get any discount by keeping my old phone… on the contrary I feel like I’m wasting money but not taking advantage of something I’m paying for.  How many people feel the same ?  I would guess a lot and this can be a reason to upgrade for many.

I hate to see people coersed to get devices which they might not overwise need and which provide huge amount of waste. Apple claims to sell some 15M of devices by the end of the year.  How many of these will really make a difference in peoples like and how much will be a new toy for a week and when used the same way as the old phone while causing resource waste ?  Many perhaps.

To be fair your phone is not the largest polutant out there yet this is perhaps one of thouse cases when you’re the most encouraged to waste.  If you’re buying new car it unlikely will be subsidized to the same extent… and also you will likely use less gas while driving it.

I find many technical gadgets to be good enough to be kept for a long time these days. They are just good enough for what I use them for and upgrade would unlikely give me anything more than couple of days of excitement of new toy.  For example I’m still using Kindle 2 for more than 3 years as well as my Dell desktop for about same time.  This would not look like extremely long use form any of you but it is for me – previously I had an itch to update my hardware much more frequently.

Where I still see improvements important for me being made is laptops.  I’ve been changing laptops about once per year resently because I dammaged and lost some, other died and I am also still looking for more – decent power to run the applications and yet enough battery life to last transatlantic flight together with desent weight.  I have not found the perfect one yet… but when I will I will likely keep it for a long while.


On Namespaces

It is interesting to see how namespaces evolve in the Internet…. I find it especially interesting the move in importance from Open namespaces to the Closed ones, controlled by specific companies.

In early nineties when WWW was just getting on its feet everything was about the Domain Name namespace.   These were issued on first come first serve basics and you would not need any approval just desire to pay yearly maintenance fee.    This free system had its flaw.  Domains could be registered in thousands (or even tens of thousands) for purposes of being resold when someone is looking for well sounding name or pure extortion when names matching other company name or product name are created.   As internet Evolved it became harder and harder to get decent names.   Somewhere in 1997 we got “virus.ru” domain name which was available – note a word which you could find in a dictionary.  In 1999 we had “spylog.com” registered which is decent sounding word combination.  In 2006 we took a while to find the name for the company “Percona” which also would be available in  .com TLD    Now with over 100million of domains registered in  .com TLD more and more people have to buy domain names from third party for anything what sounds and types reasonably.

With all its problems and limitations the benefit of it is you essentially “own” the domain name and the decision of court of law is required to take it away from you, so it is rather safe to be working on building value connected to domain name.

Continue reading

Freak out your customers. Ask them everything twice.

When I call my bank they ask me to enter my debit card number when as I am connected with associate he asks me the card number again as well as last numbers of SSN for verification purposes.  Same happens in other cases  – for example just today online Skype support would ask me to enter Skype name and email and when they ask me exactly the same stuff again.    I find it very annoying and somewhat disrespectful.

If you want to show your customers respect and what you value their time do not ask them information what you already have on file or which they already previously provided. If you need to validate something for a change you can ask some related question, for example “do you still live at Rose Rock Cir”. This is understandable and not nearly as annoying.   If you need to check something for security purposes why not to tell that, people like to see you care about their security or at least understand it.  Yet if you validate something for security purposes you do not have to ask the whole thing.  For example being asked my address for verification purposes I often would tell my house and street number yet the phone agents would often also want city and zip.  It is a waste as there are  only couple of locations in US for my street address (excluding city)  so adding city does not increase security, and Zip is completely redundant.

What you want to sell and what people want to buy

As you might have seen we have announced our MySQL Support offering which includes pricing per server.  Did not I frequently said pricing per server makes little sense ?  Yes I did.  So why are we having this offering ?  It is because how people often like to buy it, that’s why !

We all may have a good idea how to price something in a better and more fair way.  For example  it would make more sense to sell gasoline per pound, as this is what defines its energy value, yet we all got used to buying gasoline per gallon (or litter outside of US) and hence getting a little bit different energy value for our buck depending on the temperature.   Yet how do you think the gas station selling gas per pound would do ?  Probably not so good even if prices will be competitive as it is confusing for people.

At Percona we learned the same things apply to support.  First people would like to buy support on fixed budget and have us to take a risk whenever they will need to have a little help or a lot of help in the given year, so hourly caps are gone.   There is also standard of per-server pricing in the industry and even MySQL Space, check out for example Oracle or SkySQL.   We want to be be easy to do business with so if purchasing support priced per server is how you prefer to see the world, we’re ready.

Another advantage of per server pricing it allows us to make Percona Services more affordable – smaller shops having  just one MySQL  Server in production can get Support just for $1500 per year which I think is a very good deal,  previously they would have to shell out at least $7500 for Silver subscription.

For people running “larger” installations, with large being as much as 10 servers there are better options available with unlimited options.  This is because as long as you get to 10+ servers server number starts to matter even less – you just have “many” of servers with most likely many of them running similar load – either being replicas or shards.

You may also note all our support is 24/7 which we think is how support should be.  Business hours only support  makes as little sense as business hours only emergency room or car insurance which only covers you on highways.  You do not know when you may run into the problem and if you’re looking for production support you need it to be 24/7.

Does support comes with limits ?   Of course.  If you have the plan with unlimited hours you can’t have the same open ended services as we do in MySQL Consulting where we can write code for you or just log in and make your problem go away.  We define support as response to specific incident or coaching you to resolve the problem. We however can’t write your application code or re-design your application architecture as part of our support offering.

Does this mean Percona is moving away from our per hour pricing. No! It is still available and it will continue making sense for a lot of our customers looking for services truly consulting in nature. With new MySQL Support offering we’re  just enhancing your choices and making support more affordable for you.

Class Actions

If you’ve lived in the USA for long enough you surely have gotten mails about you being members of some “Class Action” suite.  These are amazing.   The email just landed in my mailbox about Dell doing something wrong with their home warranty service.  They deny they did anything wrong but agree to settle because this is cheaper than going to the court.   So what is the deal ? As a member of the suite I can submit the claim for $4 to $10 “cash benefit”, yet there is $5,368,000  for “lawyers fees and cost”.

I find there are few things wrong here.  First  the whole case is made up by lawyers as a revenue opportunity – I do not believe many people would find it worth the hassle to pursue some litigation  for $4 compensation.   I believe instead of “opt out”   it would be much better to have opt in process so  only people who really feel they are suffered and need to be represented.  I’m happy with my Dell equipment and their services are good enough and I would rather them spend money on developing new cool stuff, improving their service or even rewarding their shareholders than this kind of stuff but now the hassle is put on me to opt out of the suite. I’m all for having a system where businesses are responsible for their wrongdoing but we should not create a motivation for lawyers to litigate just for sake of enrichment.

Second I really do not like this “admit no wrong doing but decide to settle”.  I do not know what is the chance of Dell actually found guilty in this case and so if it was indirect admission of the guilt or rather just desire to manager risks of uncertainty which can cost a lot of market cap for public company among with other negative consequences.  If this is second type  – settling just to save the hassle, it is a moral hazard of very bad type – meaning  you can “racket”  businesses which have not done anything wrong with a high likehood of getting something out of it.

4 types of Bias and dealing with them

At Percona we always try to do our business in a way to control our bias. I say control not eliminate as I believe this is not possible to achieve whole 100 percent.  Human mind is wired to be biased on so many different levels. To be able to control the bias – act in a way we do not affect our bias to affect our advice to the customers we need to understand what types of bias there are.

First however let me remind again why dealing with bias is important for us.  To provide best consulting possible we need to try to step in the customer shoes and provide advice which offers best solution in their individual circumstances.  Stepping in customer shoes has its own set of challenges but, but once you’re in those shoes you can get your bias to lead you astray.  For example it may make sense for customer to continue using Windows but you’re biased against it and recommend costly platform change.

Second let me highlight there is the special breed of bias called conflict of interest which usually applies to the cases where there is a clear commercial gain from giving certain types of advice. We have to deal with a lot of  relationships in our life where conflict of interest presents and this is where the individual (or team) integrity becomes paramount.  For example in commercial medicine doctor has direct benefit of suggesting more expensive (often more invasive) treatment  and this may not be always to the patient advantage.

Third let me tell about our approach of dealing with Biases –  The general idea is as follows. Bias could be caused by our emotions/subconscious feeling which may include greed or it may be conscious feeling which just does not fill with our ethical value completely. We need to understand it and when try to analyze the problem logically – what is the problem ? What kind of solutions are ? What are benefits and drawbacks. As long as you can get to more abstract details it becomes a lot easier.  It is also important to propose several solutions which might work  in this case even if you do not position something as your first choice customer may ask more details or bring up something else to the table.  For example recommending xtrabackup for backing up Innodb database I can mention LVM backup is also an option and explain what the benefits and drawbacks are factually. Even if my value of different benefits and drawbacks are different from customer’s he gets complete picture.   It may be even though this instance is Innodb only some other instance contains large MyISAM tables and customer wants same backup solution which just works for both MyISAM and Innodb and IO performance impact may not be of concern for him.

Lets get to our bias now. Here is what I could come up with, though I do not believe this list is complete.

Creation Bias – when you have created something you would have the Bias to use that product instead of something else.  We sure feel warm and fuzzy if someone runs Percona Server, Xtrabackup or Maatkit. We have to really keep ourselves in check to ensure this is the best solution available.

Experience Bias – Similar to   Creation Bias we generally are more comfortable with things we know. And objectively there are less risks of using things which you know. However you need to ensure you understand your experience may not be same as one of the customer.  One may like Java but it is not practical to recommend it in the team which has only .NET experience 🙂

Marketing Bias –  We also often get a bias even in case we have not used the software. It may be based on Vendor –  “Apple products are always cool” or more specifically something you’ve heard read or even memorized without noticing it.  It again causes us to pick some things instead of others even if there is no direct gain for us.

Now finally lets talk about Conflict of Interest – something which we especially need to stay away from.   There are many type of conflict of interest which differ from each other from level of directness.  For example if I sell my product and get cash this is a very direct link, if however I recommend Percona Server which may in the future bring me some development hours with higher probability than if customer would use MySQL it is a lot less direct conflict of interest and it is less of the problem especially on subconscious level.   We try to avoid as many direct conflict of interests as possible. For example our software is free so we do not sell you software (this has another positive side effect of avoiding vendor lock in for customer).  We also would not have deals when we would earn commission by recommending one solution/product over other.

As consulting company we however have a bias of looking to offer solutions (both development and consulting) which include more hours.  We try hard to keep ourselves in check by asking if this is something which can be made more simple or what are alternatives to the development.  For example just today someone wanted to get  delayed MySQL Replication as available in MySQL 5.6 in early MySQL version. We told them we possibly could do backport but it may be better to just use maatkit’s  mk-slave-delay.

The good thing here is what hours allow customers to really keep us in check all the time and measure our efficiency.  As customer knows how much different solutions costs to him he is able to chose less expensive or reduce project size if it became too ambitious.

As a summary  I think we’re not perfect and there are number of cases of the biased decisions I know about. I try to ensure we correct them when we can. I also know we’re in a pretty good shape comparing to most of other companies in the industry as this is something we constantly keep our attention on.

Minimal Wage, Govermnent Dependency and Education

There are minimum wage laws in most developed countries. Is it a good thing ? Austrian school of Economics argues it is not as this means there are some jobs which are not created which otherwise would. This would  drive unemployment level down, though of course lowest paid jobs would pay a lot less than is needed for  comfortable lifestyle.

Why do people would not allow the free capitalism work allowing people to sell their labor freely for any  amount they wish ?  This is because there is an assumption of imbalanced relationship.  There is a feeling if you’re low skilled and looking for job competing with many people of the same skill you will accept job with any pay and if there is no minimum wage people will be forsed to accept jobs for virtually no pay.

Do you see the problem with this logic ? As long as slavery is gone nobody can force anyone to accept any job he does not want to.  The critical assumption made here is accepting the job is the only choice  individual has if he is to make a living.  This is not really the case. You do not have to work for anyone you can be self employed!   It is not minimal wage but your income generating abilities as self employed person should define wages for lowest paying jobs.  If everyone can make at least $10 an hour as a self employed individual when there would not be jobs paying less than this amount.

One may think you need capital to make money… I do not agree, especially on the low end you can start work with little capital, little skill, you just need energy and may be a bit of creativity.   There is a free internet access in the library and you can make some money selling things on ebay,  you can do some art and  crafts (though you may be competing with people in third world for labor cost), you can provide some service for people such as washing windows or doing haircuts for their dogs.

I’m not saying everyone has to be self employed,  on the contrary I believe for most people working for  someone else would be the best choice, however I think everyone should be ready to work for yourself if
you do not find a good job.  The feeling you do not really need a job can be very empowering.

If you think about world in this way the unemployment insurance becomes not the issue too.  If you’re  let go and you can’t find job quickly enough to avoid burning all your saving away you can get to the  self employed track.  This would look horrible for people who has made a choice early in their life to specialize, so their worth as employees is so much more than they could make as self employed. This was their choice however to do so – if your carreer is risky in this sense you may consider having excessive savings to allow you to spend more time looking for job or re-train for different industry if your job no more exists.

All of this comes from my basic belief what government does not own you anything, well, at least you should not relay on it providing anything. This goes back to my upbringing in at the times Soviet Union was transitioning to somewhat capitalist Russia and many people who were counting on Government keeping their promises and taking a good care of them were burned badly.

So how about helping people who can’t help themselves ?  There are abandoned children or people who are so sick they can’t take care of themselves.  This is called Charity and it existed well before government started to engage in it.  I belive people become less charitable and more rigid if they feel this is  government job to be a charity. “I had paid my taxes, what else do you want from me” Yet government may  not be the most efficient in this allowing for abuse with resources wasted on people most familiar with the system rather than those who needs help the most.   Moreover counting on government deemphassises  a lot of family and social functions. Parents are not counting on children to take care of them in their old age and investing in their good upbringing, it decreases children responsibility for their aging parents, it does not motivates us to invest into building relationships with network of friends and  local community to be able to count on in times of hardship.

So what Education has to do with it ? Well I belive education is defines how we view life at significant extent.  I think it teaches us to grow up and become employees but not really highlights this is only one of the choices.

Picking a NetBook, 2010 Edition

My Samsung NC10 Netbook was a good choice at the time, though as time went its Intel Atom 270 showed itself being more and more limited as web sites grew more complicated and video moving to use of more CPU intensive codecs.  Netflix started to look very choppy so some of YouTube videos.  Finally after I managed to drop it on the concrete its hard drive started to develop bad blocks and I once had to spend couple of hours recovering unbootable Windows due to bad blocks in the system area.  So it was time for a change.

That Netbook also tought me the great lesson – I know knew exactly what I wanted.   I wanted the new Netbook to be as light as previous one. I looked to get a bit larger screen with a bit better resolution.  I wanted to get faster CPU and good bettery life.    I also need it to be quality built so I can relay on it during my business travels.   One choice was to get on the Business Class 12″ laptops which were close to what I was looking for but which were too pricy for my taste, another was looking at higher end consumer class Netbook.

I ended up picking Acer Aspire TimelineX 1830T-3721.  This is 11.6″ Laptop with 1.2Ghz Intel Core i5 CPU which is powerful enough basically for anything, it also includes Virtualization extension allowing me to run Linux in Virtualbox  which can be handy.    4GB of Memory and 500GB hard drive makes it quite fitted for basically any application tasks I need.  There is no more painful waiting I had to suffer with Atom CPU every so often.

The full size keys on keyboard compared to 92% on my old Samsung make typing convinient, though I should say I do not notice much difference.    The Screen resolution of 1366×768  is a big difference from 1024×600 Now I do not have horisontal scrolling on most sites I  care and using GMail etc is a lot more convenient.

This laptop is thinner than Samsung NC10 and it weights just a bit more – 3.1 lb instead of 2.8, which is hard to notice. The Battery life is quite good.  I can get 5-6 hours depending on what I’m doing which is not the perfect “full working day” but good enough for most of my travel needs.

One downside I found is a glossy screen, though this could be easily fixed getting anti reflective screen protector.