Posted: October 31st, 2011 | Author: irv | Filed under: Companies, Programming | Tags: black, employment, entrepreneur, minority, Programming | No Comments »
After reading Mitch Kapor’s post “Beyond Arrington and CNN, Let’s Look at the Real Issues” on minority software engineers and founders in Silicon Valley, I felt that I had to respond. I think that we are missing the real point of what it means that there is a dearth of qualified African-American candidates for employment, or as technical founders. What I have found is that in Silicon Valley, the quality of fairness is crucial. Everyone living and working here in the tech industry is terrified of biases of any sort, as it indicates a level of irrationality that would make any engineer nervous. Everyone here prides themselves on detached analysis of value.
I think we are all, including myself, a black engineer and entrepreneur, missing the point. The rate of incarceration, dropping out, divorce, alcoholism, drug-addiction, etc… are all far higher for black americans, also holding true for most minorities, than the mean. No matter what else happens at the point of hiring, the number of people available to hire will suffer, in general there will be fewer potential candidates period. No bias necessary. Many of the people who would have become the artists, programmers, and brilliant business people end up washing out by getting caught in the aforementioned traps.
Why do minorities get caught in these traps? Lack of education for their parents, breeds lack of education for the children. Lack of empathy for the parents, breeds lack of empathy for the children. Data bears this out. This is bad and it needs to be fixed, before attempting to hold industry accountable for their hiring practices. While the desiccation of minority groups’ chances for success in this country is a serious problem, what is worse is what it portends for the rest of the country.
All of the above traps, are catching more and more people from other groups as well. The decay of the quality of basic education combined with the reduced capacity of many to find meaningful work at the entry level has all but stopped caste movement in our country. Just look around, how many people do you know who have transitioned from poverty to middle-class wealth who was born in the past 20 years? I can tell you, not many.
Eventually, it will not be a situation limited to minorities, the only people who will have the capability to found and or work at a high level in an engineering capacity will be the very few wealthy Americans, foreigners, and immigrants to this country who have had a quality education. Quality educations do not just happen in school either, they happen in a child’s free time. Does your kid just play Call of Duty? Or does COD make your kid curious as to how they make it so that they play COD interleaved with reading books on OpenGL?
It would be trivial to refactor the educational and social systems of our country to facilitate this curiosity, but the other problems of unstable home life, uneven attendance, huge differences of wealth in school districts, and criminally low standards for teachers as well as well as ridiculously low pay make it as impossible as bipartisan budget resolutions.
Startup founders, or people even in a position to think about creating a startup, are a minority of a minority of a minority. There are, strictly speaking, too many variables that could create the appearance of bias to be certain there is one. Only when you have minority founders showing up on Sand Hill Rd with 3 million active users per month, and a 3% conversion rate to paid, but still getting turned down left and right should we think there might be a problem. And let me tell you, if a green 2 foot alien showed up in Mountain View with those numbers, every VC in the valley would be lining up to fund them.
The most important thing for us to do now is to think about what prevents people from getting to the stage where they could consider starting a company, no matter their color, and fix those problems. Until then, it is a useless indulgence to talk about biases in the tech industry.
Posted: July 6th, 2010 | Author: irv | Filed under: Companies, Lifestyle, Programming, Uncategorized | Tags: business, corporations, engineering, Lifestyle, millenials, Programming | No Comments »
Disclaimer, I am not a millennial, I am in that strange area between generation X and generation Y, being closer to the Y. What does that have to do with the topic, you ask? It puts me in a unique place to watch the struggle of ideas unfolding between the engineers coming into companies, and the engineers / businesspeople running corporations. This is not to say that all current executives are outdated, but in many companies, they have failed to update their model of the world to match increasing numbers of their customers, and the incoming flock of engineers.
The fundamental issue is that people who have had success in the past have a hard time considering that what gave them the success in the first place is not likely to continue producing success. As an example, existing business processes for tracking hours is typically to have each individual estimate ( after the fact ) how many hours they have worked on a specific project on a given day. The current method, as best as I can tell, is for engineers to estimate via points how much time it takes to perform a given programming task and do a post-mortem if the task takes more points. But this daily reporting is eliminated, which is a better, more efficient process. It is also one that revolves around trusting the engineer.
Another example is that it is not uncommon for developers working on a project to push out information about that project to the public via Twitter. Even down to the level of code commits. For the users of that product, they can choose to follow the official company feed or they can decide to follow their favorite engineers. The concept of privacy has been diminished to a large degree in modern companies. The benefit of this is that users become partners, not only in the debugging and troubleshooting process, but also in the development and planning phases. You can find, for just about any startup, engineers posting what features they are thinking about and feedback from engaged consumers, either providing amplifications, their own feature suggestions, or strong negations about where the company should be spending its precious resources. In such an environment, extreme secrecy is a huge liability. Likewise, within corporations, keeping the status of the company, and what the customers are saying about the products from the engineers is disastrous to engineers’ morale, as well as harmful to the level of understanding of the executives as to what is happening within the company. In more modern companies, the developers are treated like the partners of the product managers and the executives.
I think most of the fallacy in this regard comes from the manufacturing metaphors that have dominated the majority of the corporate worlds’ view of software development. When I look at the waterfall method, and some of the organizational structures around engineering departments, what I believe is being attempted is to reduce development to an assembly line with shift managers and the like. This can’t really work for software engineering for many obvious reasons, but probably the most obvious is that programmers, even self taught ones have more in common with lawyers than they do with assembly line workers. Assembly line workers can highly optimize their tasks due to the extremely specific level of requirements, as well as the consistency in their tasks. Developers, and the product people working with the developers, almost never have requirements detailed enough to complete the given task. Similarly, developers have a wide latitude to perform tasks in different ways as tools, managerial practices and or technology change, which is nearly daily at this point. While most manufacturing systems change once every 20 years or so, a particular manufacturing worker can master their skills and have that be applicable for their entire career.
Attorneys are typically highly specialized, and operate with a widely varying set of rules, like software engineers, they need to parse and execute on sets of specifications ( laws ) to the benefit of the person contracting or paying them. Their interpretation of a given law may not always be standard, but if it achieves the intended goal, then they are considered successful. This interpretation in law as in software engineering is more of an art than a science. This variability in going about the job from day to day creates odd management challenges that are being exacerbated for software engineering management as the millennials come into the workforce. To a large degree, having fewer, more productive, empowered engineers is obviating the need for traditional engineering management. Of course someone needs to be accountable, but if you have small groups of developers, the group can be accountable for a specific feature. Small groups of engineers make it easier for them to triage why something went wrong and prevent it from happening again. Failure is part of the software development process, but it doesn’t have to be a destructive part.
Millennials, and their immediate predecessors appear to be very comfortable with dealing with this sort of environment, they do not seem to need clear guidelines or even a clear goal. Many software projects that utilized the “agile” philosophy, which even itself is becoming dated, typically manage the process with smaller tasks that everyone in a company seem to be involved in creating. The new crop of engineers seem to be more comfortable with the self-taught, with it being more about what you can show than what you have done. Resume’s appear to be losing their value relative to a solid portfolio of open source work and products. My advice to people in high-school and college about to enter the work force is to work on a portfolio of applications first, or contribute to some open source projects, even more than attempting to get an internship at some big company. If they can make some money off their portfolio, all the better. The teams appear to be more distributed, with wide acceptance that each individual is working on their own business ideas not related directly to the company’s goals, or product portfolio.
All of these things fly in the face of the traditional command an control structure, however I believe that it will speed the pace of innovation, and improve the overall level of developers. Smart companies will harness this multitasking and openness and provide avenues for their developers to contribute new products under a “labs” or a “demo” banner, even if they have nothing to do with the products that the company makes. These companies will not mind as one of their “labs” projects earns more than their flagship product, and will provide the creator of that product a team and budget to see how far they can go. That will rapidly become the only way to retain talent as the cost of starting a business online continues to drop. Executives at these companies will treat their developers as peers in strategy as well as in the software development lifecycle. It will become clear that this method of structuring a business is correct when not the one, but the many startups offering services begin to completely demolish the incumbents. It is going to be an interesting ride… are you ready?
A few days ago, Joe Hewitt wrote a Twitter tirade about how web development has been stifled by the glacial nature of innovation at the w3c which caused a lot of reactions across the web, even some from Google. Joe Hewitt also quit developing for the iPhone because of the App Store’s policies. I have been thinking for a while about this, and it is what makes me want to write a web browser.
The web has been good at delivering applications with zero-install, as well as presenting formatted documents. The latter is what HTML was designed for. XHTML was created because HTML was overstepping its bounds. Currently I believe, even though native developers tend to look down on web developers, that developing a web application is one of the single most difficult challenges for modern development. There are multiple languages that one needs to master, each with a different metaphor, syntax, and implementation. There is overlap between the languages. There are latency issues, networking issues, lack of resources on the client, these are all incredibly difficult things to deal with in general even with native implementations, the problems with the web exacerbate these issues.
The benefits of this approach, full on applications could be developed as one stack and would appear on the client as the developer wished. The performance would be insane, the execution environment would be simplified such that we could develop an adequate sandbox. Authentication would be up to the developer and would be native, many of the security issues would go away. One could code their application as code + data to render as pages easily.
I think that native jitted code + data mixed together are likely our future, especially watching how the App Store is taking off with consumers. Even they seem to be choosing native applications delivered over the internet over web apps. I think that this is a primitive method of what I am describing, but the benefits to the end user and for the user experience are clear.
Posted: April 19th, 2010 | Author: irv | Filed under: Programming | Tags: computer science, new york times, Programming | No Comments »
I read the NY Times’ article on women in computer science, as well as an excellent rebuttal by another blogger named Hillary Mason. I wanted to put my $0.02 cents in on the subject.
Basically computer science is hard. Are there issues for women? Sure there are. Are there issues for men? Sure there are. There are issues for everyone in choosing to become a software engineer, especially one who has a goal of building their own startup. There is going to be personal sacrifice, hard choices, and fighting to get through the glass ceiling. It is an incredibly competitive environment, and being a minority is an additional strike against you. I have found through experience that you just need to be better and work harder than everyone else. Does it suck, yes, is it fair no, nor should it be. It is just the way it is.
Creating some sort of affirmative action in the long run is a bad idea, so is segregating your potential startup from the system that is causing you problems. The goal is to integrate into the environment, not to stand apart from it. I have continuous issues balancing my family life with my professional life. Not to mention my mental health as well as needing to be continuously coding and reading to stay competitive. To claim that women can’t do this is counterproductive to the argument, plus just patently untrue.
In many companies for which I have worked, I have been the only African-American software engineer, but not because someone made a program for me to get me in, it was because at that time I was the best human being for the job. One day I hope to build a business of my own and run it, not as an African-American entrepreneur, although that happens to be what I am, but as an entrepreneur. I will sweat, sacrifice, and claw to do it. If I can’t get funding, then I will try something novel like charging from the start, and slowly building a business that is always profitable and always in the black.
My reaction would be to agree with Holly Madison, suck it up and get busy building a quality business based on your ideas. Know that it is going to be hard and that it is not going to be fair and that there will be times where you will be utterly alone and be the only one who believes in what you are doing. Know that you and your family will have to make terrible zero-sum-game type of decisions. This all comes with the territory for anyone. To complain that you didn’t come from Stanford, that you don’t have a computer science degree, that you can’t work 60-80 hours a week, that you are a minority, that you are a woman, are all just excuses. Get past them and get busy making something that no one can deny. The playing field is level enough that if you are careful and do things just right, something wonderful will happen.
Posted: February 3rd, 2010 | Author: irv | Filed under: Lifestyle, Programming | Tags: data, exercise, hardware, Lifestyle, metrics, Programming, software, weight loss | No Comments »
About 8 months ago I started running because I noticed that Nike+ was built into my iPhone 3GS. I had run in the past, but never very seriously. I started to lose a bit of weight, but it wasn’t coming off like I felt it should. I’ve always been heavy, except for the couple of times in my life when I went on a crash diet. Once I lost nearly 100 lbs, way back when I was 16 by eating every other day. Recently I think I have found something that works so A few days ago I felt motivated to indulge in telling my story on my blog since perhaps someone would find it helpful. Since my weight kept going up, and I kept exercising harder and harder, I thought that something must be wrong with me. At one point I was riding my bike to work, totaling over 200 miles a week and I didn’t lose an ounce.
Over the years, the lifestyle of a software engineer and a literature nerd took its toll on me and I gained back all of the weight that I had lost when I was 16, and then some. I signed up for one of Nike’s virtual running plans and started to run more and more. I was really enjoying analyzing the data that was coming out of my nike plus, that combined with the weather data from slowgeek. But I wasn’t really losing any weight. I just sort of settled into the fact that maybe I was just one of those people who were meant to be fat.
My wife wanted me to go to the doctor and ask for statins, which I did, because she was worried about my blood pressure and cholesterol, which I hadn’t checked in forever. My doctor agreed with me that something wasn’t right about the fact that I wasn’t losing weight, even though I was working out like a demon, that my blood pressure was so high, and that I was having allergy and miscellaneous immune issues. She didn’t want to give me any drugs, she is an awesome doctor, so she sent me over to get a lab done.
The cool thing about my doctor is that, much like a software engineer tracking a problem, she was able to string together a bunch of seemingly random data, anecdotal and otherwise, mix it with empirical data from the labs and quickly come to a working theory of what was going on. I had a chronic vitamin D deficiency. I know it sounds like a joke, and I thought it was pretty silly when she prescribed me high-dose vitamins, but my laughter in ridicule quickly turned into joyous laughter once the weight started dropping off. My cholesterol was fine, so that was the only thing. My doctor told me that a) everyone has a vitamin D deficiency, and b) since I was African American it was worse for me, especially living in the Bay Area. She said that vitamin D plays a role in, get this, metabolizing sugar. That without it your body has a hard time using the energy from sugar. Well most of everything we eat is sugar ( high-fructose corn syrup ), so this would explain why I couldn’t lose weight.
That wasn’t all either, I had been moody for quite some time, but the Vitamin D mellowed my moods and helped me to concentrate. About the same time I had been reading a book called The Omnivore’s Dilemma. I was becoming more conscious of what I was eating. I figured that since the iPhone was so awesome at capturing data with the Nike+ and that I always have it with me, if I could find something that would track my nutrients and calories on the iPhone I’d be able to see what was going on with my diet.
When I first started using LoseIt! (ITMS Link) I was shocked that almost all of my calories, between 4,000 and 6,000 a day were coming from carbohydrates. I was running at something like 90% carbs 2% fat and the rest was protein. Prior to using the program I was under the assumption that all I had to do was keep my fat down. With my Vitamin D deficiency preventing me from actually burning the massive carbohydrate load I was putting in, my body was just storing everything. Once I finally got the Vitamin D levels evened out with supplements, I started cutting down the calories.
Almost immediately I noticed that I started choosing the lower calorie foods with high protein such as eating a chicken breast, with no bread or rice, and a salad, instead of a salad and a horde of pretzels. The pretzels had an insane amount of calories, so did bread, and rice, so I stopped eating so much of those almost immediately. I started eating way more fat, and when I say way more, I mean that between 12% and 20% of my calories were from fat. I still eat way more fat than people say is good for you, all the while losing weight at between 1 and 3 lbs a week, with little to no ravenous hunger that doesn’t occur at mealtimes. Of course I avoid trans-fats and saturated fats, but I am not afraid to eat a steak or dark chicken meat. Once you get into training for endurance sports your needs will change, but at the beginning it is really good advice to follow Chris Carmichael’s : Eat Right to Train Right foundation percentages.
All this data was making me giddy, I could actually see what was happening to my body as I changed the mixture and quantity of what I was eating. As I fixed my diet, the constant hunger went away. LoseIt had me at around 2800 calories when I was at about 260 lbs, but now it has me at about 2060 at 200.
LoseIt works on a very simple assumption, that 3500 calories a week equals 1 lbs, so short 3500 calories a week, you should lose 1 lbs per week. With all the data I can now see, based on what I am eating, what is happening to me during races, long rides, etc… I have a much better understanding of why I am cramping up on rides, or during swims, or why I bonk, or can’t crank up the output on some days. I am learning which foods burn best for what.
Everyone thinks that the government has BMI and the diet guidelines all wrong, and to be fair, it is a bit off, but not as much as people think. People want to believe that they are just the way they are, and that they don’t need to, or can’t change. Many people think that they can be healthy while being seriously overweight, or that they are somehow special and the laws of thermodynamics don’t apply to them. Everyone is so focused on eating more “good calories”, etc… That is all bullshit. A calorie is a calorie. If you don’t burn it you will gain weight. Unless you are a triathlete, marathoner, etc… 3500 calories = 1 lbs period. First you have to fix any chemical or other issues in your way, but aside from physical biochemical issues, it is possible for everyone to be at a healthy weight, and it isn’t hard as long as you can be anal about tracking calories.
I would have lost 1 lbs per week had I not been running, picked up swimming, and started cycling more. Instead I was losing around 3 to 3 and 1/2 lbs per week. While doing all of this I was building my CycleMetrics application for the iPhone. Testing it was a monster, I had to do intervals to test the power output on the bike even when I wasn’t riding simply for fun. All the while aggregating my exercise data on Google Docs, and using LoseIt to count calories. Some people may think it is rude when I pull out my phone at dinner and start tallying up the damage before I eat it, but it helps me with portion control, and screw them if they don’t like it.
At the moment I have lost 60 lbs. I still have a bit to go to hit my ultimate goal, which is to hit the top end of the normal BMI range for my height and shoulder width, but now that I have a framework with which to control my weight and keep myself at peak physical and mental performance I am not concerned that I will hit them.
I would like to editorialize a bit and rant about the food industry. It is absolutely insane that I need this much technology to figure out what is in the foods that I am eating. I shouldn’t have to track every calorie this way, corn shouldn’t be in everything in the quantities that it is. The government shouldn’t tax fossil fuels, they should tax high-fructose corn syrup. Also, eating out is inordinately hard. I needed The Daily Plate ( Livestrong ) (ITMS Link) Application just to figure out how to approximate what I was eating and what the calories are in stuff at restaurants. They should really consider putting this on their menus voluntarily, and if they won’t the government should make them do it. Only because if they don’t everyone’s health care bills will rise.
With all of the money that we are looking at spending to improve health-care, I can’t believe that we don’t do something about the sugar / corn intake of people. Unless you look at what you are eating, the amount of calories that you can consume is really absurd. The bag of pretzels next to your desk that you munch on while coding and listening to glitch music is about 1400 calories, you’d be better off with two butterfinger bars. Better than that would be to eat a balanced lunch with fat, carbohydrates, and protein.
If I was reading this, and I was still where I was, I’d be saying to myself, yeah its all well and good that you found religion, but you are working out, eating right, and got quality medical care. It worked for you, but how can I know that it will work for me? Its a good question. I don’t know that it will work for everyone. However, what I do know is that for each of the people that I have given this app to who have seriously tracked what they ate, their results are the same as mine. Think about it this way, development without a framework is hard, it is tough to know where to start, and even harder to know when you have built enough.
LoseIt is like a framework, it makes the hard decisions about how much and what to eat. It lets you think about that fancy Lisp project that you want to build, instead of feeling guilty about eating too much, or wondering whether to eat this or that. Really it doesn’t matter much what you eat, if your goal is to lose or maintain a healthy weight. I still eat McDonalds, although I get a happy meal, I eat dominoes and drink copious amounts of beer. The difference is that either I work out to get rid of the excess calories or I stop eating and drinking when I hit my limit. That is not to say that you won’t have any health problems eating this stuff, it is just to say that your weight won’t be one of them if you stay within your calorie boundary.
As far as the exercise goes, there are so many different types of sports out there, even people who say they hate sports admit to not trying them all, who knows, you could have a desire to be a curling champion, or to do pole vault, etc… There is something for everyone.
If you capture the data, imagine the awesome analysis software you can write to find trends. Right now I am cross referencing my diet with the weather, and speed with altitude to see at which humidity and carb levels I perform best at with altitude. It isn’t so much that I care, it is mostly about the fun with analyzing the data. I never thought I would have so much access to the inner life of my body. Anyway, I this is about wrapped up. If you are having trouble losing weight look into Vitamin D, and if you haven’t already, grab LoseIt (free) and a copy of Omnivore’s dilemma, it will change your life for the better.
I forgot to write an additional thing that I am doing to control my weight. Since my father died a little over a year ago, I have been trying to get 100% of my fiber each day, somewhere between 25 and 30 grams. Coincidentally, I learned later that fiber plays an important role in helping you to feel full after meals, as well as helping your body dissipate excess calories. I would recommend that anyone increasing their fiber, however do it gradually or you might have some uncomfortable results.
Posted: May 7th, 2009 | Author: irv | Filed under: android, iPhone, java, mides, PHP, Programming | Tags: android, IDE, Interpreter, iPhone, mides, parser, PHP, Programming | No Comments »
For the past few weeks I have been working on writing a PHP interpereter for Mides. As I know that I am not allowed to do it for iPhone, as it would involve downloading and executing scripting code, which is not allowed. I have only been looking at the tokenizer for the iPhone, while I have been looking at the full monty for Android.
I actually have gotten to the point where the tokenizer is passing a lot of my tests, but there is a lot of ground to cover with PHP. Tommy Carlier’s blog on writing a parser has helped tremendously. Still, even after getting the tokenizer working, I have to write an interpreter to execute the tokens. This involves implementing the many methods that PHP has built in, so it will be a while until I get that completely done.
I am pretty frustrated with Mides for iPhone, actually, I wanted it to be a full PHP implementation and editor, but until the Terms and Conditions change it isn’t worth the effort to build an entire PHP interpreter just to have the app rejected. I am still improving the iPhone version, and the next update should be a huge improvement over what is currently there. I had to remove the nesting because I need the memory for documentation search as well as the PHP tokenizer that I am working on. Overall, the next update for Mides will make it better, it will be close to what I was hoping for, but it will never be completely what I was hoping for on iPhone.
The Android version of Mides on the other hand is shaping up nicely. The Android text view supports color so I have some syntax highlighting happening which doesn’t hurt performance on the G1 too badly. The tokenizer is mostly done, and now I am designing the interpreter. Development on Android is going a lot faster since I don’t have to worry quite as much about memory leaks, although if you try you can still make them happen.
At any rate that is why there haven’t been any updates for Mides in a while, I have been working on localization, parsers, and interpreters. I am spending most of my time on the iPhone version of Mides, but it seems that I am getting farther with the Android version, go figure.
Posted: March 8th, 2009 | Author: irv | Filed under: Programming | Tags: careers, hiring, Programming | No Comments »
I was listening to the Stack Overflow podcast the other day and the discussion turned to hiring. I then thought about an ongoing issue with my family, basically it is endemic to good software engineers, “Why can’t you just stop when you get off work?”
The reason finally dawned on me today. The answer is that we are hired for what we are as opposed to what we do. This is what causes problems when companies hire people who think programming is something they do. Writing software is something that I can’t stop doing. I honestly feel depressed if I have some idea that I don’t work on.
As far as hiring is concerned, it makes sense for these companies to view hiring these types of programmer in the same way you would hire an actor, or an athlete. You are looking for someone who not only has talent, but who also has a natural obsession for solving problems. IMHO talent is useless without practice. I’d take a hardworking programmer with little to no talent over a talented programmer with no work ethic 10 times out of 10.
Realistically, companies don’t need for their whole programming staff to be of this type, but their lead programmers should be. Programmers who only work from 8 to 5 will be fine implementers, but no matter how much schooling they have, they just won’t have the experience to design great software, likewise they aren’t likely to have the passion to dream up the most creative solutions for your toughest problems.
This is why it doesn’t always make sense to pay all of your programmers the same amount, unless you hire only essential programmers, Google strikes me as a company like this. These programmers aren’t necessarily paid for their output, but more for their ideas, so they might often not appear to be doing much of anything. Then suddenly they will be taken with something and will work day and night on it.
In most organizations, the essential programmers are like lead actors, rockstars, and top artists, usually they are a pain, and you may often wonder if they are worth what you are paying them, but the project just wouldn’t go without them.
The key with all of this is to not mistake essential programmers for normal programmers and vice-versa. You will have problems retaining the essential programmer if you hire them to just be a programmer, and you will not get the performance you expect from a regular programmer in a role that should be occupied by an essential programmer.
Posted: December 27th, 2008 | Author: irv | Filed under: Programming, Rails, Ruby, Uncategorized | Tags: Programming, Rails, Ruby | No Comments »
Bold Predictions About Rails
For a couple of years now I have been very disparaging of Rails, and in general the whole convention driven development movement. I have always figured that maybe I could do things a little better myself perhaps. Recently, however I have found myself using rails, and in a larger sense, adding Ruby to my toolbox, and have been a much happier programmer.
What suprises me is how many excuses I found myself making about why I shouldn’t use Rails. What I eventually realized is that I didn’t care if I could write that CRUD code a little better, or if I could shave a tenth of a second of execution time from whatever function I had written in *name the language* the important thing was that I could get my project done and get outside, or work on several projects simultaneously, instead of only two. I found myself spending time and thought cycles on code that was directly related to solving my problem, instead of code to get me to where I could start solving my problem.
Now I am always suprised when I hear my same old excuses coming out of other good developers’ mouths. I want to say, “you just don’t understand.”
The reality of the situation is that the ground has shifted under our feet. Because of rails, business stakeholders are demanding quicker turnaround on features and products, and non-rails shops are struggling to keep up.
Performance is always the first crotchety old excuse, the answer is memcache. Then reliability, f5 big ip is the answer.
Really everyone, evolve or die, it’s as simple as that.
Internet Explorer 6 Hangs with Multiple Connections
The code needs to be rewritten no doubt, there are many more efficiencies to be had, but that didn’t explain the hang. I combed over the server, watching response while a user was using the application. The map server stresses the machine, because it needs a ton of I/O and it would spike the CPU frequently, but no processes went to 99% CPU utilization, and the server seemed to respond to other clients even when one of them was hung up. It was pretty clear then that the problem wasn’t with the server. To take this logic a little further, we built a load test using wget and saving the result to a file. We looped over the calls as fast as we could and we never caused the map server to hang. It performed as expected.
The next logical step was to look at the possibility of corrupt files. We did notice that we could get the map server to crash when we fed it corrupt files, but we found no eveidence that the files that we were using in production were corrupt in any way. At this point we were plenty dejected, because we had spent something like 35 hours over a couple days working on this problem and we had nothing. We performed a new ColdFusion install on a different server, we built a server with better hardware, we reinstalled the map server application multiple times, nothing seemed to affect it. We even improved the network bandwidth available to the client, still nothing. At that point I was down to either it was the code, or it was the client.
To test this theory I commented out all of the flash calls on every page and went through the application to try to cause the system to hang. I couldn’t do it, so I had effectively limited the possible cause to the Flash movie. I started to go through what the Flash movie was doing, and what could cause it to fail. The demis people told us that they had seen hangs when the map server wasn’t responding, and the Flash player was parsing XML. This lead me to try the application in Firefox, and lo and behold, it never hung up. It worked like a charm. The only problem was that our client was set on Microsoft Internet Explorer
I started about the arduous task of removing all XML parsing from the Flash code, then I tried it and it still hung. I was truly disappointed, but I rethought what was happening with the XML. It was making server calls, I realized that I could have up to 8 consecutive connections going on. At the time I thought it was nothing, but then I started trying to find out what was different between Internet Explorer and Firefox. I happened upon an article on MSDN about a known bug that Internet Explorer will hang for 5 minutes when there are 2 persistent connections to a server, and rich content is downloaded. I had found my culprit. It turns out that I had to add 2 keys to the registry. MaxConnectionsPerServer, and MaxConnectionsPer1_0Server. I set the latter to 8 and the former to 24, hexadecimal. The keys need to be DWORD keys.
That would allow 8 connections for HTTP 1.0 and 32 or so connections for HTTP 1.1. The HTTP 1.1 guidelines recommend that there only be 2 connections allowed, but if Firefox wasn’t adhering to it, why should I. I added the keys to HKEY_CURRENT_USER>Software>Microsoft>Windows>Current Version>Internet Settings and it worked like a charm. Everything was perfect. Talk about looking for a needle-in-a-haystack. I’m still amazed that I found it.
The purpose of this entry is so that no one has to go through the week that I just went through. Generally no software should be in front of the client before it is ready, but in this case we already had a client. Hopefully this will help anyone out there who is experiencing hangs in Internet Explorer. Darn Microsoft and not fixing bugs for almost 3 years!
*EDIT Make that 8 years, since IE 8 appears to still suffer from the same problem!*
Here are some helpful links that might be better at explaining than I am…
Wininet Connection Issue
IE Hang Issue
Safari and Standards Compliance
Apple with Safari 2.0 has taken a major step toward standards compliance and largely are taking a leadership role in this area with its outstanding support for the Java runtime. I have heard some griping about Apple using KHTML, the default rendering technology behind the Konqueror browser for KDE, for a base, then running away with the open source once they have figured it out and not giving it back to the OSS community.
While I am extremely happy that Apple has made their browser Acid2 compliant, and they may have one of the fastest CSS rendering engines around built into the AppleWebCore. It is pretty upsetting that they would not share these advances with the developers working on KHTML so that it could also pass the Acid2 test. I can understand that some things you want to keep close to your vest for security reasons, but I can hardly believe that changes you have made to the way pages render in a browser could compromise your system integrity. This appears to be a situation in which Apple wants to be the most standards compliant platform on the market. This would be fantastic from a business standpoint since many in the scientific and mathematics communities would probably prefer to use technology that adhered to standards so as to better communicate information between offices, regions, and countries. I can understand that Apple wants to distinguish its platform from others, and I love the fact they are using standards compliance to do this, however I feel that it is to break the spirit of open source / corporate collaboration not to give something back to the KHTML community.
Microsoft is promising that its IE7 browser will be standards compliant, but just how standards compliant is really the question. I think that Microsoft has learned the error of its proprietary ways. Sure it will continue to bundle its software with everything anyone buys from them, but I don't think they will continue to cripple other products to make theirs look better. They seem to have given up on their own version of DHTML and are happy with XHTML. I noticed that their primary page even validates now. I think that it makes sense for Microsoft to go the standards route also, and with no shortage of developer feedback, they have almost no excuse not to.