Stuart Roseman
Office365 authentication is broken and insecure

Authentication on Office 365 email services is pretty broken and provides inadvertent account information at the moment.

If you try to login via IMAP to outlook.office365.com with an email client using a username and password combination, their reply is based on the existence of the username. They are ignoring the password.

They reply with “OK LOGIN completed” if the username exists.

They reply with  ”NO LOGIN failed” if the username does not exist.

This is regardless of the password.

If you have the correct username and password, after you login, you are able to execute IMAP commands.

If you have the correct username and the incorrect password, you will get the following error when you execute any subsequent IMAP command: User is authenticated but not connected 

This bug breaks the feedback on  every client’s authentication mechanism. Try typing the right username and the wrong password and see your client assume everything is just fine.  You won’t find out that anything is wrong until you discover that you aren’t getting your email.

And this is not the worst. The SECURITY HOLE here is that at the moment every hacker on earth can go to outlook.office365.com via IMAP and check to see which email addresses are being served there.  They give you the “OK LOGIN completed” when the username is correct and “NO LOGIN failed” when it is incorrect.

This is a public request for Microsoft to please fix this bug.  We have failed to get them interested in fixing it by the normal channels.

The greatest milk frother in the world

I very rarely make a product recommendation, but felt compelled to in this case.

My wife is a beautiful Argentine (http://danielacorte.com) which means I spend some portion of my life looking for the perfect cortado.   As many of you know, this is a double shot of espresso with a large dab of foam.

This entry is not about the espresso which of course is extremely important.  It is about the foam.  I have tried a zillion steam wands over the years and even tried whisking by hand.  I have one word to describe these experiences: exhausting.

Here is the **only** way to foam…

You pour cold milk in and press a button.  A magnetized whisk spins at high speed while heating the milk.  In the end, depending on the milk (I suggest Trader Joe’s Soy Vanilla Milk), you get absolutely perfect foam with **NO** work.

For those that care about foam and have a spare $100 lying around.  Run do not walk to buy this thing!

Here are 2 links were this is sold:

SeattleCoffeeGear

Amazon

image

Remembering my mom

My mom died 25 years ago on the Nov 17.

The reason I know the date so well is that, if you are Jewish (I am), you go to temple on the closest friday night to that date for the rest of your life.

You go to temple to participate in a thing called a Jahrzeit.  It is the remembrance of those that have died. I have been to one or two every year of my adult life and they never fail to move me.

The way it works is the Rabbi asks if there is anyone in the congregation who has a sick friend or family member that needs support this week.  Those people are encouraged to stand and speak the name of the person who is sick.  They remain standing.

After that, the Rabbi asks if there is anyone who has experienced the death of someone close to them in the past week.  They too stand and speak the name.

Then, the Rabbi reads a list of everyone that has died in past years on this anniversary.  This part is really something. Without fail almost all the names have someone to stand for them. And in an odd sense of community, I recognize most of them from our visits in years past.  

I don’t know who they are but I share this remarkable bond with them just by sharing this moment over the years.  I notice those who have grown old and now have to helped to stand but none-the-less still make the trip every year. And those that came alone before, but are now married. Or those whose spouses are no longer with them.

But, the thing that really moves me, every single time, is the next thing.

The Rabbi asks everyone to stand in order to show that no one of us is alone.  That this grief we all feel is in common.  In common, in the deepest way.  In common, in a way that no one can take away from us. We are all going to die.  All our mom’s are going to die. But somehow there is grave beauty in the knowing and in the remembering.  And a strength that none of us has to go through that alone because we are all in this together.

This ritual forces everyone in that temple to spend at least part of this day remembering that one person who left them so long ago. 

The funny thing is that now that my children are a little older, I find myself thinking of her more often.

Especially when I see my daughter stand her ground in an argument where she is right and she politely, but firmly won’t give in to the superior fire power of an adult or teacher. Whenever that happens I always think: my mom is still here. She hasn’t really gone away. My mom who went to Radcliffe back when their were still quotas for Jews.  My mom who went to MIT business school when that was still unheard of for a woman and then had the audacity to start her own business. My mom who lives on in the polite, fearless strength of of my beautiful 8 year old daughter.

Cheers to my mom.

it sane or wise to archive an email located in one of the sane folders (i.e. SaneBulk) once it has been read? Or is it wiser to just leave it there? If I want to zero in my InBox, shouldn't I do the same with my @SaneFolders when I have time to do so? I would like to understand the SaneBox philosophy and perspectives in processing email effectively

Hi,

Thanks for asking.

https://support.sanebox.com/entries/23125107-What-do-I-do-after-I-am-done-reading-a-Sane-email-

You can and should archive, delete, or file any mail you find in a Sane folder once you are done with it.  

Remember, the goal is to have an inbox of zero, and removing the email once you are done with it will help you get there

Archiving, deleting, and filing into non-Sane folders doesn’t train us.  Here is a document about all the ways to train SaneBox: https://support.sanebox.com/entries/23126606-How-do-I-train-teach-SaneBox-

Best Wishes, Stuart

Storage failures in hosting facilities seem to be a bit common these days. There is some speculation (no evidence to suggest it, mind you) that this could be related to clandestine intelligence-gathering operations (offlining for forensic imaging); do you have any comments or concerns about that possibility?

I think it’s more likely that the companies with the failing services are anonymous,  short term profit based public companies so they are cheaping out about the wrong things.  They save money this quarter but cause huge problems a year later.

The hardware we purchased to replace their “commercial” shared service is more redundant in every way, will provide superior throughput, and will be in my control.

I think we are on the verge of an era of smaller privately held companies that actually care about their service and their customers. Tell your friends about SaneBox

Some details on SaneBox’s downtime on Wednesday

In the old days, I would hold a software release for 7 days.  That 7 day rule was unbreakable. No additional non-bug fix changes would be allowed and much of the company time would be spent identifying and fixing bugs and getting the release ready.

In the world of Ruby on Rails with tests written first and code second, and the power of git to be able to pick and choose changes for release we are able to lower that release time to hours.

It is likely that most customers of SaneBox don’t even realize how often we are releasing new software.  Someone making a bug report might realize it when their fix gets pushed thru our development cycle in a matter of hours.

Usually the release causes a 20 second down time of the website and up to 10 minutes of down time for filtering for some accounts as our system is restarted.

On Wednesday August 21 at 8:45AM ET, we were making one of those releases.  This change included adding Rails 3.2 asset pipeline capability to our website.  It had been developed on a branch over a period of weeks and tested and code reviewed as we do with any other change.  

But, little did we know that this line of code deep in the Ruby gem that gets executed during deploy:assets:precompile: 

          tmp:cache:clear

was about to cause us endless problems.  It deletes everything in the Rails cache directory.  No, it doesn’t just delete things related to the pipelining or related to assets, it deletes all your cache files.  Sigh…

OK.  Big deal you think.  Shouldn’t cache files be temporary and easily replaced?  Yes. But they are there to allow the system to avoid doing expensive operations all the time.

For instance, it is necessary for SaneBox to keep a temporary cache of the meta information on the emails we are tracking.  This cache gets created when you sign up.  New items get added to it as new email arrives in the Inbox.  And this cache gets refreshed periodically.  If it should disappear or become invalid for some reason, our processors simply go back to email server and recreate it.

But it is one thing to recreate the cache for one account or one machine or even a couple of machines than it is to recreate all the cache for everyone.  By a factor of 100.

We always have spare bandwidth and cycles and RAM but not to allow a spike of 100.  And no database on earth can withstand a spike of x100.

So here’s what happened on Wednesday:

1. Our 8:45AM release wiped out the cache on every machine

2. We shortly noted an odd usage uptick off the database but no errors. Specifically no exceptions.  The system was running just fine.  Except it suddenly was running “a little” hotter than usual with a queue of queries we don’t normally see on startup.

3. We identified a sql query that was too big.

4. We fielded another release to fix the sql query.  Which we now know wiped out our cache again.  And big surprise fixing the symptom didn’t fix the problem.

5. Upon further diagnostic we realize that our cache has been wiped out. We now know the problem causing all the symptoms but not the cause.

6. We include the sql query fix in the previous tag and release that and start doing 2 things.  1) Ease the machines and database so that the cache can be recreated for all accounts.  2) Figure out what it was in the release that deleted the cache.

7. After some mysql configuration adjustments and various software changes to stop processes from being reaped prematurely, allow mysql backlog algorithms to operate under this new load, we were making some progress when our main mysql instance hung (we have filed a bug report for this).  

8. We had to bring the database down and up.  A process that takes over an hour.

9.  At this point, not wanting to risk another database hang, I decided we would bring up one machine at a time.  Let the processing clear and then bring the next one up.  We have a lot of machines so this was not an easy choice to make. But again we started to make progress.

10. While all this is going on the other half of the team is scrambling to find that evil line of code and design a fix.

11.  Oh yes, and some of the apache processes on some of the web serving machines got into an odd state because of the database issues so they needed to be restarted from scratch.  But since it wasn’t all of them that were having problems we needed to debug that as well.

12. We had many customers fully filtered by 3PM and finally had all customers fully filtered by 7:15PM.

Not a great day.  Because of that one line of code we inconvenienced many SaneBox-ers from all over the world. And I think my engineering team and support team has aged an additional decade.

I do personally apologize to the entire SaneBox community.  This is exactly the kind of thing that should never, ever happen.  Each production release requires my sign off.  So, yes this is my fault. And it is now my job to make sure it never happens again.

For those needing a silver lining: we did release the new software the next morning without any new surprises and the website is now 2-3x faster with a zillion additional little and big improvements. Oh yes, did I mention that Rails 4 asset pipelining doesn’t wipe out your entire cache!

So, to the two people that made it down to this paragraph, you are asking yourself two questions: 

1) What was it in our development process that allowed this to happen

2) What can we change about it to avoid this ever happening again

I am working on that and will report back.

RIP Lewis Rubin

I’m off to a funeral this afternoon in Lexington MA, the town I grew up in.

Sadly a very kind man just passed away.  Lewis Rubin was the father of some childhood friends.  He was also a physicist and entrepreneur but mostly I knew him as one of those rare fathers who seemed to truly welcome the chaos of modern family life.  His was one of those homes that were a refuge for friends and family alike.

It is a shock that he is gone. Over the years, he seemed to remain unchanged even as everyone around him (myself included) got older and older and older.  I hadn’t seen him in several years but my last memory of him was no different than those from 35 years ago. So, hearing that he was gone seemed almost impossible.

I realize now that my experience is a personal one of childhood dinners and school events and runs and vacations that happened a million years ago.  Since then, I’ve graduated from college, had jobs, started companies, gotten married and had my own children.  

I guess I’m sad that I never got to tell him how much those memories meant to me or that I fully intend to be the same kind of father he was.  I guess the only thing left to do is to honor his memory by being kind to my children’s friends and in turn welcoming and creating some of my own modern family chaos.

Lewis Rubin died of cancer on July 15, 2013 - he won’t soon be forgotten.

When is it time to close your startup and try something new?

You are most productive when you can focus. Don’t kid yourself, the hobby startup that you have been pushing along for years, is taking up creative time and effort and money that could be better spent making something, anything else be successful.

There is no crime in joining a team.  All those out there in one man consulting shops pushing their startup along an inch a day, consider what you could be accomplishing with a well managed team pushing toward a finish line that is attainable.

Please assume that most of your productive time is spent on your day job.  Or at least assume your boss thinks that is true.  Or more importantly know that your team mates are treating the day job as their path to the big time.  They are thinking about ways to make it successful in the shower.  What are you thinking about in the shower?

Some businesses are evergreen. I founded http://gamesville.com in 1995 which we sold to Lycos in 1999 and it is still running and making money.  So I don’t mean to say all businesses have a specific lifetime.  I am very proud that 13 years later our economy still has a nice place for that business.

But, I’ve also started other businesses over the years that were not Gamesville.  In my experience many of them had their time and place and more importantly there was a distinct moment when the world moved on and it was time for the founders to move on too.

This is easier to say than do. Maybe turning your back on a past business is more difficult than moving on from a failed personal relationship.  Especially when you feel that you gave birth to the business.  

I certainly commiserate with how hard this can be.  When I pivoted Visible Certainty to become SaneBox, I was leaving 3 years of 24 hour a day work and 1+ million dollars of development but digging that hole deeper wasn’t going to make the world a better place.  I wrote down all the reasons that that project didn’t work which was very purging:

http://stuartroseman.com/post/619953720/out-with-the-old-business-in-with-the-new  

I highly recommend that all the entrepreneurs within the sound of my voice do that exercise.  Why has your startup not succeeded?  

At the time, I left visible certainty up and running for another year I felt a very strong personal bond to the users.  We stopped actively maintaining it. But the software was pretty amazing and it just continued to run and run.  At least until we need to upgrade the OS.  Which brought into stark relief how much of a distraction it could be.

In the end, when I really did shut it off, the world did not come to an end. Instead of hate mail,  I got a series of very sweet emails from former users.  And my world became a better place with less unnecessary distractions. It felt good. It felt like a rebirth.  I couldn’t really move on until I had buried the past.

And here is the thing.

Your startup, your hobby, the thing that isn’t succeeding, can’t be more important than your day job.  You want to be known for something.  Use your skills to build something that you will be remembered for.  Such effort requires focus and dedication.  So long as you can’t give up on the last thing, you can’t move on to the next one.

Here are some rules about how to know it is time to move on:

1) There were going to be plenty of other ways to provide the same service for free or the same price that you are selling it for

2) it was never going to be big enough because it isn’t yet big enough

3) the opportune moment had passed and now the big public companies are all releasing capabilities similar to the ones you were hoping to provide.

Lastly, remember that you have crazy skills.  When you tell the story of the Internet to your grandchildren you should be able to point to one shining success that was created by those skills.  Be able to say that you waded into the stream and pulled out the gold with your bare hands.  Be able to point to the evergreen business that was built by your hands.

If this is you and you are ready to move on, SaneBox is hiring.  Come work with us to make the world safe for email.  This business does scale.  This business is evergreen. This business is crazy fun.  You want to be able to point back to this time and say “Yup I built SaneBox - it was a crazy fun ride!”.  Send a resume and a cover letter with at least one joke to roseman dash hr at sanebox dot com

ronnkliger:

Well, that’s it! After 26 years in the corporate world I’ve made the plunge. I’ve left a comfortable, secure, well-paying job for the unknown. I should be questioning whether this is the right move but I’m really not. That must be an indication that it is indeed is the right move or that it…

It’s never too late for a fresh start!

SaneBox bounce rate increase caused by Firefox 10.0.2 and fixed by Firefox 11.0

For anyone out there that has seen their bounce rate go up after Feb 17th and been chasing red herrings trying to figure out what changed.

Firefox 10.0.2 caused the SaneBox homepage bounce rate to go from 25% to 90% for our FF users.

Here is a snap of our GA analytics spot lighting browser == FireFox  

https://skitch.com/stuartroseman/8ju9s/site-speed-google-analytics

Here is a snap of our GA analystics spot light browser != FireFox (and yes something happened between March 7th and 8th - but one thing at a time)

https://skitch.com/stuartroseman/8jw81/site-speed-google-analytics

The annoying thing is that you can parse your GA data a bunch of ways and get a similar bump.  Interestingly we thought it was an international routing issue at first since certain countries had the the bump and US appeared not to.  But that is a dead end probably caused by the up take of Firefox in certain places and specifically how much of our traffic from that country was from FF users.

When we figure out what is actually causing this I will add to this post, but in the meantime I thought it was important to get this piece of information out. Since googling for this was not helpful

Update: 3/20/12 our FF bounce rate is now receding dramatically with the release of FF 11.0 - I’ll post a chart when the trend steadies out.