Skip to main content
Nov 17, 2014 2:39 PM UTC
Question

Software Testing / Software Quality

Since this is a relatively private forum, maybe we can have a frank discussion about software quality. This is an area where PTC is really falling down on the job. Maybe Creo Direct is great. Maybe Mathcad is pristine. But Creo and Windchill are rife with problems that any reasonable software testing regimen should catch before code is ever released to the public.

How can software be released when there are issues with saving or exporting a file? Example? Creo 2.0 M080 could not save or export a mechanism file. Oh, the message dialog box said the file saved... but, in fact, it does not. This is infinitely repeatable. Your only option, move to M090. Of course at M090 there are other problems. Saving the assembly used by your mechanism results in the file being dumped into your working directory (even if you're working within Windchill and are currently attached to an active workspace). How does this get through?!

So your next best bet is to move to M100... which is God-awful SLOW compared to M080 and M090.

So your next best bet is to move to M110... which cannot exit a sessionwithout complaining "Cannot exit. Background operations are occurring...". You must kill the xtop.exe process with Windows Task Manager to exit the program.

So your next best bet is to move to M120... or M130...

What. In. The. Actual. Hell. Is. Going. On.

How is this a defensible position? How can PTC ask customers to keep upgrading and upgrading and upgrading? How can customer support be so surly that their first line of defense is always to blame the customer? And the second line of defense is always to claim our problem is that we've not UPGRADED (again!).

It's one thing is we were running Wildfire 4.0 or 5.0 and complaining about bugs in the software. But when you're running the absolute latest Windchill 10.2 and the bleeding edge Creo 2.0 or 3.0 builds, how can it be necessary to continually upgrade to achieve even a basic level of stable software operation?

NASA seriously does not have the time or resources to dedicate to having people run around all day upgrading people! Not to mention, the software requires elevated privileges to install - which is increasingly causing problems that PTC sweeps under the rug. I won't even mention the system security issues in Windchill.

But before I scatter this mini rant too widely... let's refocus on software quality. There should be a team of testers and a test script. Someone should have to sign off on any changes made to the code. Simple things like interoperability between Creo and Windhcill should be a primary focus. But more than this, the absolute basics must be covered... like verifying that files are saved where they're supposed to be!

I feel like I've been a beta tester for this software for 25 years. It's really getting old. Major software vendors simply cannot crank out professional software without a strict, regimented, and thorough test plan. There's no way this is happening. Corners are being cut... this must be the case. If that's not true... then there are simply too few testers or they're signing off on code that wasn't actually tested. If you plop 1500 test scripts on a tester's desk and he's supposed to finish them all before code is released, what stops him from just signing off? If there's a deadline and he's going to catch grief from management for not completing his work on time, what prevents him from signing off and hoping that no one ever notices he never actually tested the code?

There is a problem somewhere... a big, big problem.... and it needs to be addressed.

If anyone has any feedback, comments... or even criticisms of my assertion that software quality is lacking, let's hear them. If we're not here to make a difference, what's the point, right?

    5 replies

    Nov 17, 2014 3:20 PM UTC

    I don't think Mathcad Prime has any more bugs than would be expected in a complex high-end software package. But perhaps that's because badly needed new (that is, new for Prime, old for Mathcad 15 etc) features are implemented at such a grindingly slow pace there's not so many things that can go wrong. If Prime 3.0 can't even save a file in Prime 2.0 format, it can't do it incorrectly! If 3D plots only have a miniscule number of options, what could go wrong? If there are no embedded controls available, they can't have bugs in them!

    So perhaps the solution to the Creo problem is:

    Creo Prime!

    Creo Prime 1.0: basic 2D sketching capability, like AutoCAD 1995

    Creo Prime 2.0: Wireframe 3D modelling, like AutoCAD 2002

    Creo Prime 3.0: ????

    Nov 17, 2014 3:39 PM UTC

    Working with PTC software often bears striking similarities to being in an abusive relationship. PTC keeps beating me - yet I still stick around.

    Eventually you have to look in the mirror and ask "Why do I stay here"? I guess my answer is "The next software company will probably just beat me, too... so it's better to stay here where at least the beatings are consistent."

    But seriously, the problem with software quality is indicative of virtually everything today. Is it just me or does it seem that no one can do their job anymore? I spent so much time cleaning up after mistakes other people have made that I've come to expect incompetence from everyone. I find myself absolutely flabbergasted, speechless and left searching for adjectives to thank someone when they perform even a modesty competent job.

    And perhaps I'm just too uptight and intolerant... but it occurs to me things weren't always this way. Quality is no longer a buzzword ... and it's certainly no longer any sort of top priority. This doesn't just go for PTC - it seems to go for just about everything.

    Nov 17, 2014 5:35 PM UTC

    Brian, I absolutely share your feeling on every level of your well thought out comments. I too find personal pride in ownership for a job well done absolutely lacking in the workplace today at every level. This is, unfortunately, the generation we raised. Sad indeed at every level of interaction I have with the world in general. Common sense has been handed over to the machines.

    As for the software performance... there is good reason I am -still- on M040. Even with bug fixes in later versions, finding a stable version is a real issue. I actually follow your lead when you post your opinions. Haven't heard if you actually rolled out M120 yet. I simply cannot afford anything that stops my work dead in its tracks.

    But the real problem you highlight, and failed to mention, is that the flaws in development are leaving many in the cold when they cannot afford maintenance. I would dearly like to hear that these "bugs" or shortcomings are not something tolerated to make sure people continue to pay up for maintenance. This is obviously likened to a ransom when you simply cannot stop at a stable version. If this business model has crept into PTC's thinking, I happy that retirement is not far off.

    I would request a change in the support culture at PTC to make a commitment to the customer; You will get -every- maintenance release for your version until the end. At one time, an annual maintenance contract would get you a new version each year. But now, with a 1/2 year slip on Creo 3.0, this is no longer the case. I had maintenance for Creo 2.0. I should have support to the finish of Creo 2.0 without being held to another year of support. Yes, it is a cash cow, but with the poor performance of early and interim releases, this is highway robbery. when you buy the software, you should have the right to a stable release without having to pay extra. After all, maintenance is an option, not a requirement! In other words, even if its a new offering, provide a version maintenance contract! I want Creo 2.0 to the end of its development. A Creo 3.0 upgrade will be extra. This is the way every other major software does their upgrades except CAD. The current system works well for major corporate accounts that simply want the option to remain up-to-date. But us individual contributors, this is simply lost productivity and expenses that take a huge bite out of my bottom line.

    Although testing is a huge issue, I want to add the fact that it has come to light on the forum that there are a lot of very old SPRs initiated in WF that were never resolved even though we are now in Creo 3.0. I suggest someone at the upper level of management pulls the report on just how much development is truly lagging behind. If I were quality, I would push for the policy that all SPRs will be resolved for the new version release. That will complete the old version (see note above) and start the new version with a clean slate. I do not see anyone making a clear distinction in the -performance level- of the new version where old engineering issues are simply passed along.

    I participate in the forum to know what the software can do, cannot do, and what may have completely missed the mark in specifications. I want more reliable and better software. And just like you, Brian, I do not want to be the beta tested. I paid good money for maintenance.