August 27, 2010
Kyle Brandt, a system administrator, asks Should Developers have Access to Production?
A question that comes up again and again in web development companies is:
"Should the developers have access to the production environment, and if they do, to what extent?"
My view on this is that as a whole they should have limited access to production. A little
disclaimer before I attempt to justify this view is that this standpoint is in no way based on the perceived quality or attitude of the developers -- so please don't take it this way.
This is a tricky one for me to answer, because, well, I'm a developer. More specifically, I'm one of the developers Kyle is referring to. How do I know that? Because Kyle works for our company, Stack Overflow Internet Services Incorporated©®™. And Kyle is a great system administrator. How do I know that? Two reasons:
- He's one of the top Server Fault users.
- He had the audacity to write about this issue on the Server Fault blog.
From my perspective, the whole point of the company is to talk about what we're doing. Getting things done is important, of course, but we have to stop occasionally to write up what we're doing, how we're doing it, and why we're even doing it in the first place -- including all our doubts and misgivings and concerns. If we don't, we're cheating ourselves, and you guys, out of something much deeper. Yes, writing about what we're doing and explaining it to the community helps us focus. It lets our peers give us feedback. But most importantly of all, it lets anyone have the opportunity to learn from our many, many mistakes … and who knows, perhaps even the occasional success.
That's basically the entire philosophy behind our Stack Exchange Q&A network, too. Let's all talk about this stuff in public, so that we can teach each other how to get better at whatever the heck it is we love to do.
(Sometimes I get the feeling this idea makes my co-founder nervous, which I continually struggle to understand. If we don't walk the walk, why are we even doing this? But I digress.)
The saga of System Administrators versus Programmers is not a new one; I don't think I've ever worked at any company where these two factions weren't continually battling with each other in some form. It's truly an epic struggle, but to understand it, you have to appreciate that both System Administrators and Programmers have different, and perhaps complementary, supernatural powers.
Programmers are like vampires. They're frequently up all night, paler than death itself, and generally afraid of being exposed to daylight. Oh yes, and they tend think of themselves (or at least their code) as immortal.
System Administrators are like werewolves. They may look outwardly ordinary, but are incredibly strong, mostly invulnerable to stuff that would kill regular people -- and prone to strange transformations during a moon "outage".
Let me be very clear that just as Kyle respects programmers, I have a deep respect for system administrators:
Although there is certainly some crossover, we believe that the programming community and the IT/sysadmin community are different beasts. Just because you're a hotshot programmer doesn't mean you have mastered networking and server configuration. And I've met a few sysadmins who could script circles around my code. That's why Server Fault gets its own domain, user profiles, and reputation system.
Different "beasts" indeed.
Anyway, if you're looking for a one size fits all answer to the question of how much access programmers should have to production environments, I'm sorry, I can't give you one. Every company is different, every team is different. I know, it's a sucky answer, but it depends.
However, as anyone who has watched the latest season of True Blood (or, God help us all, the Twilight Eclipse movie) can attest, there are ways for vampires and werewolves to work together. In a healthy team, everyone feels their abilities are being used and not squandered.
On our team, we're all fair-to-middling sysadmins. But there are a million things to do, and having a professional sysadmin means we can focus on the programming while the networking, hardware, and operational stuff gets a whole lot more TLC and far better (read: non-hacky) processes put in place. We're happy to refocus our efforts on what we're expert at, and let Kyle put his skills to work in areas that he's expert at. Now, that said, we don't want to cede full access to the production servers -- but there's a happy middle ground where our access becomes infrequent and minor over time, except in the hopefully rare event of an all hands on deck emergency.
The art of managing vampires and werewolves, I think, is to ensure that they spend their time not fighting amongst themselves, but instead, using those supernatural powers together to achieve a common goal they could not otherwise. In my experience, when programmers and system administrators fight, it's because they're bored. You haven't given them a sufficiently daunting task, one that requires the full combined use of their unique skills to achieve.
Remember, it's not vampires versus werewolves. It's vampires and werewolves.
Posted by Jeff Atwood
The chasm between developers and sysadmins is really nothing more than a responsibility boundary issue (i.e., who gets to wake up at 3AM to fix something) which has grown out of proportion in way too many companies, unfortunately.
Fortunately, people are starting to notice how this can affect negatively their companies -- check out the Devops movement (not associated with them, though I find their ideals really aspiring):
Agreed. I'm a programmer by trade, but I tend to get along better with system administrator people. Weird, huh?
I also feel that people should develop a wide range of skills spanning programming, system administration and project planning. This tends to give people more empathy for the other "roles" in the company.
As a former Sys Admin, and Aspiring Developer (So that makes me Michael from Underworld? an Abomination). Why do you think you need access to production servers? and what level of access do you think you need? The werewolves are (were?) the daylight guardians of the vampires.
When both the developer and the sysAdmin are excellent at their jobs, there is no problem.
The real problem is crappy sysAdmins getting in the way of good developers or viceversa.
It was an excellent presentation, the video doesn't seem to be linked yet on the code4lib pages but volunteers have been adding them to the Internet Archive (possibly elsewhere).
Sadly looking at the video now it seems the slides didn't turn out very well, but the slides are on the previous link.
There is apparently debate on the supernatural archtypes of each side ;).
Most of the time developers need just read access to production. This is to allow them to know that the code is really the code they expected. I have had a hard time getting even that.
Write access just leads to accusations that something was changed at 3 AM. Not having write access to production, and the separation of duties is a nice point for audit as well.
Note: I work on large financial systems, which is gamed, could pipe money to me. Not having that ability is good in that it removes suspicion that anyone is doing it.
I think you can't be the best in either field without mastering the other, personally. The best programmers are expert systems and networking people, and the best systems people are expert programmers (not just scripters, programmers). To truly excel at either job, you need a deep understanding of the other.
Once you reach that point, it's really just a matter of which job role you've signed up for (and I do believe that separation of job role concerns is important). Back to the original point of the article: Sysadmins letting developers fool around making changes in production (in non-emergency scenarios) is about as bad as developers who think making occasional ".bak" files on their hard drive is an effective form of version control. It's really bad practice.
Production systems shouldn't even be fooled around with by sysadmins. They should be automatically deployed and version controlled (puppet, chef, etc), as should all of the deployed code. The only reason anyone (developer or sysadmin) should be logged into a production machine is to deploy a set of tested changes via puppet, or to investigate unique production support issues (in a readonly fashion if at all possible). This is why we have dev/qa environments and test suites. Any scenario you can present which seems to justify a manual hack in production is in fact just an indication that you have more development and testing work to do (in the developer's code, or the system's configuration) back in your dev/qa environment.
To avoid problem I think that both of them should collaborate since the very first steps of the project. Most problems derive from the fact that developers tests software in production environment only in the final phase of the project. This lead to all sort of annoying problem, and in the worst case lead to installation of Visual STudio or similar development tool in production machine to understand what is wrong.
To minimize this, each project sould have a Continuous Integration machine that periodically deploy the project automatically in production like environment (Usually QA). If you handle a project in this way you will face less friction between developers and sysadmins, because problems arise one at a time, and there are a lot of time to resolve avoiding "manual hack". Having a CI installed forces also developers or sysadmins to write script to automate the installation to production environments, and this greatly simplify life of everyone in the team.
I wonder how cloud computing will affect this issue.
Theoretically, if a company hosts all its custom development in a cloud, it would need less hardware, and thus, fewer sysadmins.
The cloud computing model also plays into the original question. It demonstrates that developers don't need access to production hardware to get things done.
As a developer, I have a ton of respect for sysadmins who are good at their jobs. Unfortunately, I've worked with many sysadmins who were clueless (often the case at small businesses where the IT guy is also the sysadmin, except doesn't know what he's doing), in which case I think it's necessary for me to have root access to production. When I get the chance to work with a knowledgable sysadmin, it makes my life so much easier and I'm happy to not touch their server.
I'm happy to leave the management of the box and the network to our sysadmins, though less so when I find that I'm unable to do simple tasks. My beef is with those, fortunately few at my place of work, sysadmins who seem to think that keeping the machines running is the point of the exercise and not simply a pre-requisite to the real work the system is doing (my app). "Yes, I know that those three VMs all have low CPU usage and it would be easier to manage just one rather than all three, but that's not a good enough reason to introduce otherwise non-existent service dependencies. I'd rather that my billing service not go down just because your licensing service went wonky."
The best sysadmins, and developers for that matter, are the ones who remember that the end user is the real customer and think about their requirements from the user's perspective and not with a purely "machine"-centric mindset.
Another aspect which, perhaps, blurs the lines is the role of an application administrator. Application administrators often need more leeway than developers with respect to system access. Often I wear both hats -- app admin and developer -- on the same project and if I know my app isn't behaving properly and I need a system reboot or IIS reset to clear it, I'd prefer to be able to do it in the situations where my sysadmin isn't readily available. The best scenario in that case is to have the ability but also to have a defined protocol for what I will and won't do with it and when.
All-in-all both sysadmins and developers need to learn to trust one another to do their jobs and put the customer first. I'm lucky to have a group of system (and database) admins that get this and are nearly always :-) a joy to work with.
I disagree with the developers as vampires notion. It reinforces the dangerous stereotype that programmers do better when they stay up late. Dangerous because most (read: neurotypical) programmers are better off keeping normal hours-- and all programmers should try to be well-rested, regardless of their preferred hours.
When I've had a really good night's sleep, I'm at least five times as productive as normal. When I haven't slept well, I'm more likely to add bugs than to fix them. And if I work late, there's an event horizon after which I can save time by going to sleep.
There's a lot of scientific research on the effects of sleep on attention/focus. Programmers would do well to keep up with it. Especially this: just as the first casualty of war is the truth, the first casualty of sleep deprivation is the part of your brain that can notice sleep deprivation.
Jeff, could you expand a little on what access Kyle has to code source (TFS/SVN/etc), by means of comparison?
Jeff, love ya man, but it seems to me you wimped out on this one. You wrote a lot of words, but concluded with there is an art to it and can't we all just get along sentiments.
These are wishy-washy. Sure, they're true, but so vaguely and universally true that they don't help anyone resolve the specific issues on the ground.
I'm not trying to say you need to specify that devs always have some specific level of access to the production environment - of course situations differ and so solutions must also differ. But to really move this discussion forward, we need to start thinking algorithmic - what are the factors involved? How does increasing or decreasing factor X move us closer to or farther away from devs having full access to the prod systems?
Theoretically, if a company hosts all its custom development in a cloud, it would need less hardware, and thus, fewer sysadmins.
Sorry Todd, your theory is bad, but your conclusions are fine. I know because I'm in an org that is three years into an internal cloud system.
Systems administrators do not spend their days playing with hardware.
Now, this may be true in some humpty organizations that go cheap on the hardware, or stick their equipment in an unventilated closet next to the boiler. Most of us buy gear that will last and at least stick it in a space with a fan or two.
A component breaks, the system sends you email. You setup an appointment with the vendor, yank out the old part, shove in the new. It's all hot-spare stuff so nobody is affected.
The days of having to obsess over hardware, having to babysit servers so they don't melt are over.
What we do is deal with systems and how they interact with each other. Cloud does not reduce the complexity of those interactions, it makes them even more complex.
It requires a methodical mind, troubleshooting skills, a willingness to get out of one's comfort zone, to synthesize. It requires - God help us - a system administrator who can code, for a lot of interactions are mediated by logic and scripts.
This is a change from the way things have been. Managing servers by hand, never a good idea, will be flat-out unsupportable. Guys who even try that will be fired for incompetence.
We saw a similar change over about the time that the title 'LAN Administrator' went extinct. People who made comfortable livings running a LAN for a department or two (Think NetWare 3) suddenly had to operate at the organization level. If they couldn't they found employment elsewhere. Your LAN team went from a herd of guys to a small team.
As the LAN Administrator of 1998, so goes the system administrator of 2010. In the future there will be less of them, they'll manage more 'stuff', they'll be more skilled than their run-of-the-mill peer today.
No human should have access to production systems. All work should be done in development. This should be pushed to test systems. Once QA signs off on the code changes, then the code should be pushed onto the production boxes from test, with a plan to roll back if there are any issues. All changes should go through these steps, because nobody is perfect, every change should be carefully monitored. And of course if a production box goes away you should be able to just set up new hardware and push all your code onto the box in a few minutes and be up and running again.
Heh, this is a very interesting question. I think that it depends on why they need the access. If they need the ability to tell why applications are failing etc... then I think that they should be able to access the production systems to view that information. Or if they need examples or sample data to see what is happening on a daily basis. Granted I do not believe that they should have unlimited access all the time; but if there are issues going on with their application(s) and they need to see what/why is going on then they need to have that access.
I currently work in a position where I am in a sysadmin/datamining/developer rule. Our development group programs the application that we utilize inhouse to provide services to other companies. Our sysadmins control the systems that their applications reside on and many times they/we have to defer to the developers to let us know why something might be happening.
In our line of work our developers are working on application (as opposed to web application) level work (Java and C). The main problem for us is emulating the situations that we have; 10billion insert/update/delete (that's billion with a B for EACH of those SQL types) per day. These numbers are staggering and are over a distributed system across the internet; what makes it very interesting is that, we cannot duplicate this type of input. We don't have the hardware nor would we want to buy an extra 4-5k machines to duplicate this.
So in our environment, it is important for our developers to SEE how the changes that they have made or are going to make will change the systems performance. Even adding a single line (although this goes for almost any system it seems like ours is especially important) be drastic performance changes and even client impacting changes (I've seen issues where someone made a change that they didn't even think would be big and in the code it wasn't; and it changed the entire way we had to utilize the system from that point forward and not in a good way).
This is, of course, my view being in the middle of all the different sections. I am a user of the system that I am a developer of and I am one of the people trying to keep it running on the system that is built by our sysadmins.
Alkampfer, you are dead right. The only reason developers need access to production is when they want to hack in last minute changes that weren't found during testing and QA. If you have a proper process then these situations won't arise.
I am a programmer who thinks that developers should NOT have access to production. Even beyond that, I think the application deployment/patching should be done when developers are not even in the room. Let me defend this with the reasons I believe it.
#1. The "handoff" to the sysadmin team is an important step that causes documentation to be not only written but USED. I can't stress this enough. It is easy to "write" documentation, but the only way you can check if your deployment checklist is correct is to have someone ELSE, without you sitting nearby to handhold to USE the documentation you wrote... in most cases, the first try won't work -- but the documentation will get better (use a wiki!) as you go. I also recommend on larger teams you have different members of the sysadmin team do the deploy so you don't end up with pockets of knowledge.
#2. Beyond the benefits of documentation and corporate knowledge, it places the responsibility and control with the person who gets the 4AM call when the site goes down. Developers simply put a patch or deploy "in the pipe" and the sysadmins can decide when to deploy that patch... they can do extra testing they want, whatever, it puts control in the right hands.
#3. It creates a need for a robust and real development environments, which is vital and creates higher quality products. Developers work in a good clone of production, because they require it! It forces a good behavior.
#4. It allows developers to not have to worry about what happens after the work is done. You develop the product, you put the lotion in the basket and lower it down to the sysadmins, and move on to the next product (or feature).
~ Basically, giving access to production to developers is lazy, and it encourages bad documentation, poor testing, and binds them to a single environment they shouldn't be thinking about... developers should live in development / beta environments, never in production. If you aren't doing this, IMHO, you are doing it wrong.
I kind of like the "call for collaboration" at the end of the post. Albeit I think that it is also an attitude of the people: you don't necessarily HAVE TO give them more to do or they would be bored. If they get bored, they are doing something wrong. If they have more time on their hands, they should probably use it to study, and grow.... maybe learn a bit more about what the other guys are doing. Maybe that ITPro who is scared or scripting should try to do it and see that it won't kill him if he does... it might actually open his mind.
I generally agree with many of the comments up here: Brandon Black, as well as bdunbar to quote a couple.
You can't be a good programmer if you dont' know how systems work, and you can't be a good sysadmin if you can't code a little (or a little more).
The slides mentioned by ejunker are also an awesome example.
Albeit you only have limited time in your days, you should be striving to increase your knowledge and know more of both worlds. Because, in fact, I think that the whole division between those two worlds has been historically a marketing artefact - a strategy to create audiences to "target" and to seel products and services too.
In the unix world, you have "programs" that do complex and useful things, which really are big PERL scripts - written by sysadmins, to solve system administrations's headaches.
In the Windows world the division is a lot more visible or frequent... but lately a lot more people understood that those two audiences must be talking to each other and uderstanding each other more. Think of powershell, for example - I see it as a step in that direction.
bdunbar is also making a good point, about the evolution to the cloud, and the higher level of skills required.
the whole "division" between the two becomes a lot more blurry over time. Which is a good thing. Let's hope it eventually fades away.
Sysadmins are programmers too. They might use different languages, and their aims and jobs might be different, but they are programmers too.
Even people that design computer hardware are programmers. Chip designers at Intel probably know more hard Computer Science than the majority of people at StackOverflow.
The difference between so called "programmers" and "sysadmins" as that their roles and aims differ. They both write new software, they both worry about performance and stability, they both worry about security.
The ones performing sysadmin tasks, however, stake their reputation and paycheck on "the systems are up" and the "client-feature programmers" stake their reputation and paychecks on the wide adoption of their feature by business end users (business because of internal budget allocation, direct payment, or advertising revenue).
If you don't think sysadmins are programmers, you've never worked with good ones. I say this as a client-feature developer who sysadmins his own systems (passably at best).
Manual-override.blogspot.com made some very good points.
When I started in IT in the 80s, the IT department had 5 people in it and we did the lot (we even laid the cables). So I found it odd to be isolated from the production systems when I worked in bigger companies with separation between SysAdmin and Development. Especially when I had to jump through tons of hoops to get a change made.
However, all the points made by Manual-override.blogspot.com are very valid. It's better not to be able to hack away at the production system at 3am, as it prevents suspicion when something breaks, and any credible company should minimise the people who can access production data.
That said, it depends on the size of the company. Nowadays I tend to run my own systems (web sites), and work with small clients, and generally I don't have any SysAdmins to control anything and I have to do it all myself. It gives me some freedom, true, but I miss the furry little blighters (and yes I have been coding all night).
I'm installing MySQL this weekend. :(
I need to know who the zombies are :-)
I think in most IT departments at companies they do not let developers access production, instead they have to make a productions request of some sort to get anything deployed. This in principle is good.
However, the biggest issue is that QA on the code itself often is non-existant. People do functional testing but not code testing. In that case it matters not whether you directly change something in production or deploy it using a strict process, either way the code goes through.
The Zombies are the users. They do not know or understand what goes into making their applications neither do they care or want to understand it. They also continue to create bugs even when it isn't a bug instead it is just something that they didn't know worked in a certain manner. "brains" becomes "bugs"
Yup Suroot, is right that users fit the Zombie analogy in this ridiculous fantasy, because they don't know anything other than what they are told and they, quite rightly, have no interest in our stuff either, other than how it helps them achieve their job in the day to day running of the business.
However you have to expect them to report things they don't understand, and it's up to the professionals to be professional about it and fix it if it needs fixing, improve it if it can be improved, or identify and support enhanced training if that's the resolution required. We have to accept they don't share our fascination with our art, and not take it personally if they reach the conclusion that something doesn't work when in fact the problem is that they don't understand it which is probably due to a lack of training, or possibly because the system is too complex or not intuitive. Life would be easier without users but then we wouldn't get paid.
bdunbar, I apologize, I should've been more specific. I was positing the scenario of a company opting to host its enterprise systems in an external cloud. As cloud platforms mature, a lot of smaller companies might look to them as a way to control costs. Pushed to its logical conclusion, perhaps internal production environments will one day become an unnecessary liability.
My company migrated from SharePoint 2003 to 2007. We allocated 3 servers for it, spent a huge pile of money on licenses, and another pile to have our sysadmins and developers install, configure, and customize it. Now MS has come out with yet another version. I don't relish going through this again.
But suppose the next version of SharePoint is implemented as a set of separate (but interoperable) services on Windows Azure. If companies could migrate to "SharePoint Azure", we wouldn't need nearly as much production hardware. And if the "Software-as-a-Service" model takes off (hosted externally), internally maintained production environments would slowly become a liability. Let somebody else worry about it. All that stuff was just a means to an end anyway. And if I can get there without it, so much the better.
I'm aware that sysadmins do a lot more than manage hardware. But the focus of this discussion is about the intersection between sysadmins and developers: the production environment. So my question is: Could external clouds some day make internal production environments obsolete? Would this eliminate one of the main intersections between sysadmins and developers?
Wonderfully said, I guess that puts me (that I am both sysadmin and dev) in the abomination kind :)
As a developer who also spends a lot of time helping out with deployment of software to different environments, I feel that in most cases developers should not have any access to production. If you can't do a proper deployment on your identical acceptance environment, you should not deploy at all.
And if something goes really (involving in house code, which is extremely rare) wrong I want the sysadmin to be there and we should be looking at production together.
So if programmers are vampires, and sysadmins are werewolves, what are test engineers?
Forgotten about, I suppose.
I do agree, programmers are vampires! But I admit, I get along with our sysadmin quite well and he's definitely a werevolve. Maybe it's because in the first weeks at my company I was kind of a handyman for administration issues. But I'm not that kinda Daywalker like Vaida Dan described himself...
While dev access to prod is always an interesting and lively debate, in many situations the question is moot, as governmental regulations, such as Sarbanes-Oxley strictly limit developer access. As someone who has worked largely in the health care and financial industries I, as a developer, have almost never had more than read only access to production. And in many instances, I haven't even had that, except for the occasional temporary access for a couple of hours here or there to troubleshoot a production problem that the admins couldn't fix on their own.
Solved that one, I became both sysadmin and developer. Now the biggest problem is keeping myself out of production...
As a Developer (in terms of HR's interpretation of my job description), my position on Prod is that I should always have as much access as I need in order to fulfill my duties. I do a tremendous amount of troubleshooting of live code, and am expected to support an application's behavior in a live environment, so I expect at least read only access to the data and the logs.
Where things get particularly frustrating is when I'm "politically" responsible for an app, but a certain component is someone else's "technical" responsibility, and he or she isn't carrying the appropriate amount of weight. I hate to sound like A Jerk, but if you can't do it better than I can, and you're not inclined to learn how, then just let me do it.
This is, of course, a two way street: If my app is a resource hog and requires much bigger iron than a better designed app would; then an administrator should certainly be able to hold my feet to the fire. :-)
Why do developers need some sort of access to production machines?
I've worked in environments where they don't and it was a miserable experience.
When the developer has no access to production they have to forward all of their requests to sysadmins or DBAs.
Trying to debug programs becomes almost impossible if you have to forward a request for a DBA to read a permission, set a permission, change a procedure, etc.
The developer inevitably ends up waiting for the sysadmin to complete the task before moving forward. The sysadmin gets upset with the developer for wasting his time.
I would end up waiting for days, or even weeks for the request to come back as the DBAs had work of their own to do as well.
Then after thinking it couldn't possibly get any worse it did.
A new IT manager was hired who decided all requests had to be formally requested as a "change request" and be entered and tracked in a database.
Don't want to give programmers access to the prodcution system then fine, debug it yourself :-)
Funny how an article about sysadmins and programmers ends up in a few reader comments noting users as dumb, uneducated, and stupid. Guess it's the only thing we both seem to agree with. It's also the very thing we are both wrong about.
As for the article, as a programmer myself, definitely agree that programmers shouldn't have access to production servers. Ideally, in-house code should even be implemented on production servers as any other 3rd-party code; i.e. built and packaged. And any in-house project maintenance should follow the same procedures for 3rd-party code through more or less formal downstream bug reporting.
But lack of personnel, usual high maintenance requirements of in-house code, and always very short response times requirements, mean this "ideally" is rarely feasible. Programmers end up having to shortcut their way into the production servers for faster identification of causes or implementation of patches/fixes, or sysadmins end up having to script their way into a responsive and functional program while they wait for a proper fix from the developers.
So, I look at Jeff's post more as a warning. A gentle push. A reminder that, while not entirely possible under most circumstances, we still should always try to move to an environment where programmers stay away from production servers and sysadmins don't mess with development/test servers. The sin is not in failing to accomplish this, but in getting used to it.
By my experience the co-operation sysadmins and developers works best if they are put together in one team. Either as a release team or a project team.
Because if they work close together they start to understand what the each of them is doing and why there is sometimes a gap in between duration of tasks.
The simply reason for that is, that when they sit together in one room working on exactly the same scope they will start to talk with each other. And perhaps they will also share some time (like lunch or even some after-work event) with each other.
So, mutual respect will grow automatically.
Of course this will not work of all kind of people ... because you will still find some lone-wolves, who you do not want to miss because of their skills.
But it is alway worth to try :)
I just wanna say im happy im a vampire, vampires are so much cooler then warewolves
The secret is to work with them, not against them. You as a programmer have a need. They as a sysadmin should be able to satisfy that need in a way that is acceptable to them. That usually means finding what the real need is, and setting up a process for it.
For instance, blatant open access to one of the assets they control is not going to happen. But common tasks that you need to accomplish on that asset can have a process created.
I have basically the same issues with ServerAdmins and Support Reps. Our shop is so screwed up support reps that barely know how to use their PC with XP/Win7 are allowed to remote into our server environment to run business applications and make configuration changes!
Microsoft doesn't help this mentality with "the server looks just like my desktop so it must be the same". I really wish WIn 2003 and 2008 server was totally command line driven so users had a huge learning curve!
Man, I don't get it. I think this post might be just an excuse to compare developers and sysadmins to cool creatures like vampires.
I have to echo other sentiments shown here by saying that comparison to mythical creatures is a little lame, especially if your argument comes down to 'it depends'. And - in any event - you have missed a trick somewhat by not realising that your relationship with your co-founder is also one of these issues. Dare I say it; 'Sales' is surely another mythical creature (perhaps a bugbear?) that is sure to try us, and yet is a required part of life for many developers.
You may find it interesting to read such work as 'Six Thinking Hats' by Edward de Bono, and perhaps some stuff on Myers Briggs Personality Analysis (though I am a little sceptical). If you really want to categorise people, why not go so far as to read up on the Enneagram or Astrology... or What Color Is Your Aura? (I jest)
You can probably summarise the body of these works as 'appreciate the way other people are'; and that although it can be challenging, teams made up of different types of people (in respect of the way they think and act) can be more effective than if a team is made up of just cautious and conservative programmers.
Nothing wrong with us balding vampires, enjoyed the post- Vampires (Programmers) versus Werewolves (Sysadmins).
The reason to give access is because a bad sysadmin is a process bottleneck.
The reason not to give access is that programmers break things.
There is one essential rule for each role:
Sysadmins: Lead, follow or get out of the way.
In other words, don't be a roadblock.
Take a proactive role in each project, or be prepared to do pretty well whatever they want.
Programmers: Bad planning on your part doesn't constitute an emergency on mine.
In other words, your project isn't the most important thing in the world, and don't expect a sysadmin to jump through hoops to help you. Get Sysadmins in at the planning and design stages so you know what IT problems are pending.
I say why not? If your programmers tend to break production on a deployment then you need better programmers.
I would say developers are "Artists" and sysadmins are "Curators".
@Matt Langley, I agree. It strongly depends on the size of your company. in small companies, sysadmin role is normally given to a highly responsible person of the team, able -and like to- to monitor and prevent any server faults and to scale hardware as necessary.
However, both scenarios are very different. and the sysAdmins meadows and programmers mountains need very different kinds of persons too. so at the end you will need to train and keep with you somebody dedicated full time to the task, for the sake and peace of your company -and the mental healthy of the sysAdmin ;).
I completely agree that developers should not be given access to production systems. A solution or system architect, depending on what vernacular you're using, is a completely different story.
As someone operating in said role, it's exceptionally infuriating having to deal with a client's operations group that generally has poor experience operating servers. For example, my current customer's operators thought that having 0 free disk space was okay. Nevermind that virtual memory couldn't be allocated. The same group also found it appropriate to arbitrarly change file permissions on a server which broke the application in question. Oh, and the brilliant production DBAs threw tempdb on C:\ (RAID-1) which brought the site to a screeching halt. All of this on an application that generates $10s of million in revenues per day. Whoops.
So as a system admin there are a couple of things that can be done to help those that sit in the middle or completely on the dev side sleep at night.
First - be competent and be open about what's being done to servers.
This rarely happens unfortunately (in both aspects). Referring back to the file permission issue above (which was causing a CPU spike) - the operations group absolutely refused to believe it was something they did, in spite of numerous performance charts that could pin down exactly when something happened. So instead, developers spent 2 weeks scratching their heads and wasting a lot of money until ultimately someone remembered that they did something when they ran an analyzer on the server. Seriously - they blew probably upwards of $100k because someone didn't write something down and fill out the proper paperwork.
Secondly - just make the information available to us!
There are big boy and girl toys like SCOM and OpenView which can give everyone insight into what's going on without requiring setting up some absurd scheme to allow us to remote into boxes with some form of read-only access. More importantly, those tools let us know that it's about to hit the fan well before it actually happens and can capture knowledge to resolve issues in the future as well as trend tracking which is invaluable. Spend some money to save many orders of magnitude more in the future by investing in these tools, preferably in conjunction with some sort of profiling capability (e.g. AviCode, dynaTrace,etc...).
On the development side of the house...
Construct your applications to be operationally friendly. Instrument them. Provide proper event log entries (i.e. don't flood it with 10k entries / sec of useless crap). These all play well into being able to determine what's failing in a system - your lack of ability to account for this is your own fault and further reinforces the fact that you should not be given access to the system (SAs aside in the event of emergencies). Create run guides. In short - be professional, competent developers and architects. If you publish all the necessary information from your application, it's generally nothing more than admins capturing it and publishing it through one of the aforementioned tools. If you don't, you have nobody to blame but yourself.
Crown-Sat is a China based professional hi-tech company engaged in R&D, production and distribution of consumer electronics products, such as automotive, office and daily using electronics. As one of the best manufacturer and wholesaler in Shenzhen, we had good relationship with our customers all over the world. With the faith of “making friends before doing business”, we had almost exported our goods to the overseas market such as USA, Mexico, Germany, France, UK, Spain, Portugal, Sweden, Russia, Japan, Singapore, Malaysia etc.
"High Quality, Competitive Price, On-Time Delivery, and Good After-Sale Service" are our principle. We have received good reputation and support from our customers, promoting our development in these fields as well. Up to now, we have provided a large quantity of digital products such as the DVB, mini projector, solar related and cartronics, and the most hot sell products are Openbox S9 HD PVR,SkyBox S9 HD PVR ,Dreambox DM500 HD,
OBD China is a high-tech enterprise, which specialized in developing, manufacturing and marketing car related products in automotive aftermarket.
We have an excellent R&D team which is made up of the most distinguished experts and engineers with years of experience in this industry. All our products are manufactured with the strictest standards to meet international quality and environment standards, including FCC, CE and RoHS.
The main products of OBD China include Auto Diagnostic/maintenance Series, Auto ECU Programmer, Auto testing equipments, Odometer correction/ Mileage adjust and Auto Electronics Series etc.