Hot topic – ISO 17024
You obviously feel very strongly about ISO 17024 and auditor competence - IRCA director Simon Feary’s article on the subject in the last issue of INform (click here) has elicited a lot of response. See some general responses below or click here for responses to Simon’s open forum questions
The commentary on the impact of ISO 17024 on auditor certification makes interesting reading. There is a case to made both ways, however to make that case does require honesty in the reporting of opposing views. Feary mischaracterizes the RABQSA International programme, and several of his statements are simply incorrect. The ISO 17024 process in use by RABQSA doesn't 'break with traditional ISO 19011-based certification model’ because there is no such thing. ISO 19011 is a guidance document on the performance of audits which contains some commentary on the experience required of auditors. There is nothing about auditor certification in this document.
He goes on to say: ‘Forget, they say, input factors such as training, academic qualifications, work experience, numbers of audits performed etc. These are regarded as having negligible relevance. Only an examination of the application of skills and knowledge, independently conducted, can be accepted as a true indicator of competence.’
These statements are completely false. The truth of the matter is that the RABQSA certification process is that the four legs of individual certification still exist - the qualification, the work experience, the audit experience and the quality (or EMS) experience. They are expressed slightly differently from the IRCA/IATCA wording but are essentially the same.
The fundamental difference, therefore, is in the examination of competence. This is divided into two elements, theory and practice. The theory side is the equivalence of the written test at the end of an L/A programme with one difference. The IRCA examinations, at best, each achieve about a 75 per cent penetration of the criteria which they are supposed to examine - and the 25 per cent which is missed is different in each paper. My colleagues and I did this assessment when as chair of the Technical Committee I was working on the common examination. The changes which IRCA made to accommodate ISO 9001 did nothing to improve this level of penetration.
The RABQSA model requires that the student be examined and found competent in the knowledge of 100 per cent of the requirements. One has to ask: what's wrong with that?
The practical competency is completely new, and this is where the additional expense comes in. This is an independently witnessed audit activity which is assessed against a defined set of criteria. Again, the student has to be examined on 100 per cent of the criteria and found competent. Once again, what's the problem?
The facts differ radically from Feary's characterization of them. His next paragraph beginning ‘what makes the debate lively’ is the crux of the matter. He hints at the fact that the certification bodies generally having more interest in money than in the integrity of third-party certification and aren't prepared to take auditor competency seriously. He also hints at questions behind the validity of the accreditation process. This is the real crux of the matter. The official structures of third party certification don't take the integrity of the system sufficiently seriously and therefore won't push the issue of auditor competency.
No one at RABQSA International believes that the current model is the perfect solution - but there is no doubt that it is a better method for assessing competency than the twenty-odd year old model currently in use. Let us by all means have the debate, but let it be through honest comparisons.
Rod Goult
I wish to congratulate Simon Feary for very true and correct analysis of auditor competency and auditor registration issues. It is obvious that methods currently used for evaluation and registration of the competence of auditors are not perfect and majority of certification bodies do realise this.
This is why certification bodies have established their own comprehensive training and competency evaluation procedures. Such procedures are expensive but auditors are the biggest assets of certification bodies and they must invest in their training. The current IRCA ‘traditional' approach to auditor registration is not perfect, but the cost is reasonable for the purpose we use it, ie independent verification of auditor’s qualifications, training, experience and minimum competency required as a prerequisite to commence auditing for certification body. We also understand that comprehensive training in internal procedures and approaches should be applied for all auditors joining certification bodies.
To say that the 17024 standard will automatically improve auditor competency by introduction of an independently conducted examination of the application of skills and knowledge is naïve and misleading. The additional costs to pay for such doubtful benefits are not justifiable. Clients of certification bodies are not prepared to pay more than they pay now.
To be able to establish the reliable evaluation methods requires much more than just examination, psychoanalysis and a witness audit. I believe it is impossible to force auditors to behave the way examiner requires or expects. The behaviour of the auditor under observation or examination also differs in accordance with a lot of factors: situations observed, auditees’ knowledge and readiness for the audit, technology and industry sector, culture of an organisation he or she are auditing etc. It is important that independent examinations and witnessing are carried out by competent examiners in correct environment.
I do not believe that there are not many incompetent auditors around, and that the constant references to auditor incompetence are not correct. The main problem is incompetent use and management of competent auditors.
Mr Feary is correct in referring to the ‘certification bodies’ practices and the (lack of) control exercised by the accreditation bodies’. If the auditor is not appropriately matched with the client (ie does not understand client’s processes, technology and culture) and is not prepared for an audit, the results of this audit will not satisfy client and certification body.
I am not against principles of ISO 17024 and believe that that it may improve competency level of auditors. But I have a lot of concerns with application of ISO 17024-based auditor registration programmes to the full time auditors who are employed by certification bodies where appropriate systems of controls are already in place to ensure that the auditors they use are competent, because of: the duplication of witnessing already conducted by accreditation bodies; confidentiality issues; auditees’ acceptance of independent examiners; additional unjustifiable costs.
Alex Ezrakhovich
ISO TC176 co-convener of ISO 9001 Auditing Practices Group
The issue of determination of auditor competence and the methods to verify it should not be reviewed in isolation. Auditor competence is a very important (but not the only) component of a system that begs for reform in order to increase stakeholder confidence in management system certificates. My impression is that staggered deployment of documents such as ISO 19011, ISO/IEC 17024 and ISO/IEC 17021 does not add to the solution. We need a suite of documents that are intertwined, coherent and not a patchwork solution.
The responsibility for the determination of auditor and audit team competence lies with the certification bodies. Such entities, as mentioned by Mr Feary, might rely, or not, on personnel certification schemes implemented by organizations such as RABQSA and IRCA. I believe that RABQSA and IRCA must fully engage not only in the certification bodies, but accreditation bodies as well, and even more importantly, the actual users of the management system certificates for an open discussion about auditor competence expectation from the users perspective.
Without consistently delivering assessment services that buyers are willing to rely on, as part of their supplier risk mitigation strategy, this process will be doomed. I believe that, as pressure builds in the market place, towards the commoditization of management system assessment services, the auditor competence component needs to be seriously re-thought. I believe that more emphasis needs to be placed on auditor’s ability, intellect and courage to face senior managers. But that would come at a price.
Sidney Vianna
Director of aviation, space and defense certification services
DNV Certification, CA
I believe that the current system works but is flawed in some areas. Competence based on auditors being able to fill out forms and get someone to say that they carried out an audit does not in any way reflect the auditing competence of the auditors. I am working in Kazakhstan and over the past three years have carried out 87 audits in this country – mainly ISO 9001 but also some integrated system audits for ISO 14001 and ISO 18001 – we obviously use contract auditors from within country who we train and then evaluate in the field at our expense.
When new auditors apply to work for us and they have an IRCA auditor registration, to me it makes no difference to the way that they are trained to those applicants that do not have IRCA registration. I recently did a survey of non-conformances relating to certification audits and different consulting companies who had implemented the systems. The company where most of the consultants received IRCA registration from second party audits showed the highest number of non-conformances by average per audit by third party certification auditors in companies where they had implemented systems.
Our database currently shows an average of 3.8 non-conformances per audit in the certification process in Kazakhstan - the average of this particular consulting company is 7.4. Looking at two other consulting companies where some consultants have IRCA registration through a variety of audits consulting/second party/third party the results are far different – one consulting company 3.0, the second 4.0, pretty much on line with our data base average.
So competency based auditor approval is essential to maintain the effectiveness of the auditing process. I also believe it is part of our business to train, and we should include this in our overheads to create resources to meet not only our internal professional expectations but also to meet our customer expectations. I have seen some auditors show competence after only three audits – one observing, one selective auditing and the following full system auditing (all under guidance), others have taken seven to ten audits and others never make it.
I think there is also benefit in further splitting the competency criteria for auditor/sole auditor/lead auditor. As an example with our three-audit auditor mentioned previously – I would be perfectly confident of that auditor’s ability to take part in an audit providing that there was a lead auditor on the team, I would not expect that auditor to go out and do it alone as a sole auditor. After two or three audits as a team member then, providing that the feed back from the lead auditor was positive on each occasion then I would consider that person for a sole auditor role.
In the developing world such as here in Kazakhstan there should be an IRCA approved auditor that audits auditors on IRCA’s behalf, possibly as an overhead for IRCA, that way IRCA’s reputation relating to auditor competence can be maintained. I would think that this need not be a full time employee of IRCA unless there was a need for a recruitment drive in that particular area, but more a qualified contract auditor who lives in the area and knows the customs/diversity/culture of that area. It also means that the approved auditor could be paid at local rates, not international rates, so keeping IRCA costs down. This does of course provide some issues relating to independence and unbiased checking of auditors, but I am sure that could be overcome with the right competence criteria for the selected ‘checker’.
Roger Willmott
MD certification and training Kazakhstan, Moody International LLC
IRCA’s current certification program for auditors is valid and produces excellent results at a cost that makes sense. Another layer of legislation (of demonstrating competence through an independent examination of skills) apparently says that registrars can not be trusted to validate their auditors’ competence. The market place will decide if registrars use auditors who are not competent. IRCA provides a service of initial qualifications and continuing qualification at that level. Registrars maintain the qualifications at the local level. Since IRCA was originally established to support certification bodies, continuing on the same track with the same requirements makes sense.
Hold your ground, IRCA.
Jay Fisher