Jump to content

Recommended Posts

  • Replies 141
  • Created
  • Last Reply

Top Posters In This Topic

Hi to all,

Wow, 60 views on this thread 80 attempts at the exam and no comments, that is not like you all [:D]

Thanks to all you have had a go at this exam I hope that you found it usefull, I would love some feed back on it from those who have taken it, at the moment it appears to be pretty tough which is what I intended, but the stats show that a significant % of takers are only missing by a small % here are some raw stats to see how you compare.

I will leave the link open until next Friday

Total Exams Taken: 80

Average Grade: 54.66

Number of Exams Passed: 18

Average Passing Grade: 78.78

Number of Exams Failed: 62

Average Failing Grade: 47.66

Regards

Gerry

Link to comment
Share on other sites

Having a low passing rate can mean several things, each question needs to be evaluated for its specific performance. It could be that a question is not performing well (Everyone is answering it or a low percent are answering it)for several reasons. It could be a poorly written question; Wrong answers; Distracter answers are written improperly; The question could have a poor sentence structure; It is a trick question; and several other factors.

Over time if the poor performing questions are tracked and replaced with new questions that have been used as sample question and are performing well then the exam will improve. This takes time and many, many folks to take the exam.

Link to comment
Share on other sites

Hi Scott,

yes, you are quite correct, that is one of the reasons for allowing non-members to have a go at it at this stage, we have looked at the questions on a case by cse basis over the last few weeks to weed out poor performing questions, and either edited them or replaced them with others. but as you say the best way to analyse the performance of questions is to check the responses.

Interestingly, last week we had to re-write or replace 2% of the original Beta questions, dispite having had them previously audited by the whole question setting panel. also we took the time to make some modifications to both the system and the type of questions based on previous feed back.

i find the analysis of this exam and it's components very interesting, and will shre some further Statistics with you all when I have pulled the link down, but one that I can share now is that the weakest area is consistently "structure" which is not surprising concidering that this is also the area of most E&O claims against home inspectors, I am also disappointed to see that Chimney scores are low concidering the liability and safety issues involved.

Regards

Gerry

Link to comment
Share on other sites

A few questions:

After being subject to years of spam from Nick Gromico, NACHI's owner/founder, why would I want to fill in the required fields with all of my contact info to try the test?

Why does Gromico reply to requests to not receive any more spam with nonsense and profanity?

What percentage of the failed tests are existing NACHI members?

Link to comment
Share on other sites

Bill,

It is a little early to do much analysis of the results, but there appears to be very little difference between the results of the NACHI member results, and those from outside of our organization, as I said I will post more details after next Friday when I have a better handle on the stats.

In very rough terms about 22% are passing the exam, with about 35% failing by a small margin.This I think mirrors somewhat the stats of other exams where the first time pass rate is somewhat small, but the subsequent attempts (normally after further study) are more successful. One of the major goals of this exam is to help inspectors to analyize their own weaknesses, and to that end the back-end reporting shows how a respondent has faired in each of 10 catagories.

Regards

Gerry

Link to comment
Share on other sites

Originally posted by Gerry Beaumont

Hi to all,

Bill and Brian, I have e-mailed to both of you a username and password that I have set up so that if you wish you can look at this in a "spam free enviroment"

If anyone else has the same fears but would like to look at this please e-mail me.

Regards

Gerry

Gerry,

I'd be interested in taking your test if I could do so without risk of spam. If you'd be so kind, you may e-mail me at jim@benchmarkinspections.com

It seems from your description that you're trying to recreate the NHIE. Why bother? They've already invested hundreds of thousands of dollars in that project. When it comes to testing a home inspector's knowledge by using a multiple-choice question/answer format, they're the king of the hill. If a NACHI member wanted to distinguish himself, he could go take the NHIE today. Your new test will always have a "me-too" aura about it.

If you want to provide your members and the profession at large with a meaningful alternative, why not go in a different direction?

Show them a picture illustrating a defect and give them 60 seconds to write just what they'd write if they'd encountered it in the field. Then judge their response: Did they comprehend what they saw in the picture and communicate it effectively, writing in a style that would be acceptable to an educated client? Or did they try to cloak ignorance with big words and hustle?

In the next picture, show them two defects. Then show them multiple defects that interrelate with one another.

You'd only need about ten pictures to separate the wheat from the chaff.

- Jim Katen, Oregon

Link to comment
Share on other sites

Hi to all,

Jim, the initial Beta version that you will see is somewhat like NHIE, however the second phase of it is technologicaly so far ahead of NHIE that it ain't funny, your idea of using pictures and interpreting the visual information within them is already in the works together with some pretty advanced systems to support this style of question, but for now baby steps I have to validate the system before I make the next change. However if you do have a go at the exam you have a 50% chance of seeing a visually based "test" question that is in their to check the system.

Regards

Gerry

Link to comment
Share on other sites

Hi to all,

Just a quick up date, with thanks to those who have taken the time to try this out, and especially to those who have made meaningfull suggestions.

Total Exams Taken: 111

Average Grade: 53.73

Number of Exams Passed: 28

Average Passing Grade: 79.46

Number of Exams Failed: 83

Average Failing Grade: 45.05

As you can see about 25% are passing this exam at this point, and I can see from the stats that the weak areas are about the same, interestingly the results this week leaving it open to non members are about the same as with our original beta test group (only 2% higher pass).

I have today loaded up about 20 picture based questions to play with, now that we have validated the basic system, it will be interesting to see how these questions perform with the added benefit of visual stimulation.

This exam will be left up until next Friday for all to play with.

Regards

Gerry

Link to comment
Share on other sites

Originally posted by Gerry Beaumont

Hi to all,

Jim, the initial Beta version that you will see is somewhat like NHIE, however the second phase of it is technologicaly so far ahead of NHIE that it ain't funny, your idea of using pictures and interpreting the visual information within them is already in the works together with some pretty advanced systems to support this style of question, but for now baby steps I have to validate the system before I make the next change. However if you do have a go at the exam you have a 50% chance of seeing a visually based "test" question that is in their to check the system.

Gerry,

Thanks for the safe access to your new exam. I'm not sure what you mean by "visually based" questions (Aren't they all visually based?) but I didn't get any with pictures.

The questions I got were very similar to every other home inspector exam I've taken. Very ho-hum. Some seemed irrelevant to what we do and some, in my opinion, lacked the correct answer from among the choices. I'd very much like to see what you could do with pictures on this test. So far, it's nothing more than a me-too attempt.

- Jim Katen, Oregon

p.s. My overall score was 88%. I scored 100% in three categories, ventilation & insulation, interiors and (oddly, since I've always thought it my weakest area), fireplaces, stoves & chimneys.

Link to comment
Share on other sites

Hi Jim,

thanks for the feed back, I don't know when you took this exam, but yesterday (Sunday) I added about 50 picture based questions to the exam pool, to start de-bugging that system, it is not perfect yet but some of the visual questions invite the respondent to study the picture and select potentialy more than one answer based on what they see (and would need to report) from the picture.

If you have the time please take it again and you will get an idea of where we are taking this system.

Regards

Gerry

Link to comment
Share on other sites

Originally posted by Gerry Beaumont

Hi Jim,

thanks for the feed back, I don't know when you took this exam, but yesterday (Sunday) I added about 50 picture based questions to the exam pool, to start de-bugging that system, it is not perfect yet but some of the visual questions invite the respondent to study the picture and select potentialy more than one answer based on what they see (and would need to report) from the picture.

If you have the time please take it again and you will get an idea of where we are taking this system.

Regards

Gerry

OK. I took it again. I recognized some of the questions from my first attempt, but there were several new ones this time including several with pictures.

The good news is that my score was almost identical - 87 this time instead of 88. I take that as a sign that the test is consistent.

The picture questions provided an interesting diversion but didn't expand the breadth of the test in a significant way.

For those that haven't seen them, and without giving anything away, they go like this: there's a picture of a thing (say, a lobster) and a question, "This is a picture of a _____." Choices would include, crab, lobster, starfish, flounder.

Another version shows a picture of a defect that you might see in the field and is followed by several descriptions of the problem. You can choose as many of the answers as you feel are appropriate to the picture.

Both of these types of questions are fine, but they don't make the test stand out from the NHIE.

- Jim Katen, Oregon

Link to comment
Share on other sites

Hi Jim,

thanks again for your input, the initial picture questions that I've loaded are some what simple, as I did not want to push the system to far without validation, but I think the inclusion of real inspection issues will set this appart from other tests (heck when I took NHIE the only visual questions were some realy naff diagrams, that were one their 50th photocopy reincarnation) The ability that we have to include real reportable issues, and test members ability to report on multiple issues, is I think a major step forward.

Regards

Gerry

Link to comment
Share on other sites

Yeech! I got a 68. Of course, 20 points behind Lord Jim after 2 years in the field isn't the end of the world, but still...

Like Jim I thought some weren't very relevent and some didn't have what I thought to be the correct answer in the possibilities. A lot of it seemed more like a code test than anything else, which is natural to some degree, but I usually look that up where and when I need it. I don't know enough of it though, which is why I joined the ICC. I'm sure I missed a lot of the questions about certain heating systems too. I studied all of the above before the NHIE and passed it, but in the time since I've forgotten the bulk of it because I never see anything but gas or electric forced warm air here. Criticize me if you will, but I'm not going to spend time reading-up on oranges when I work in an apple orchard and there's more to know about them.

So Gerry, how is this to be administered, online or in person? Is it an open book test, 'cause I assumed it wasn't and didn't lay a finger on a book or Code Check? If it's done online you would have to consider it open book, I think.

Alright my imported colleague, I've obliged you with public candor, so kindly oblige me likewise. I'd like to hear your thoughts on Mr. Gromico's (?) unvarnished efforts to undermine other state and national tests by compiling the test questions for NACHI members to peruse. As one who is responsible for compiling such a test, do you feel such actions are appropriate or ethical?

Brian G.

Link to comment
Share on other sites

Hi Brian,

thanks for your in put, yes this exam is a little "code based" at this time as we used a lot of Code references to validate many of the original questions and answers, having said that is it a bad thing when alot of what we report on ulimately are code issues, anyway that aside, as you will no doubt have seen i have started to include a lot of picture based questions and multiple answer questions which will change the feel of the exam somewhat.

As to how this is going to be administered, that has not yet been determined, there are many options open to us, and that desision will ultimately be made by our members, for now it will be open for our members to use to help them understand their weeknesses, and act as an exam prep for state licensing exams.

As to the question of Nick's NHIE quote he misunderstood the question sources that we had available, yes many questions have been reported as being from NHIE but that is only to say that we collect questions from many sources, not the least of which is from members (and my students) that have taken that exam, and every other training org does the same thing, I have very many crib sheets in my posession that were prepared by the training schools from exactly the same sources.

Regards

Gerry

Link to comment
Share on other sites

Originally posted by Gerry Beaumont

...yes this exam is a little "code based" at this time as we used a lot of Code references to validate many of the original questions and answers, having said that is it a bad thing when alot of what we report on ulimately are code issues...

I think use of some code is perfectly valid. I hope the day will come when HI's need a code certification for any state license (in a fair arrangement). The question here is how much, and which parts are genuinely relevent. I thought some of what was used was not relevent and well beyond the normal scope; some other was well-chosen (IMHO).


As to how this is going to be administered, that has not yet been determined, there are many options open to us, and that desision will ultimately be made by our members, for now it will be open for our members to use to help them understand their weeknesses, and act as an exam prep for state licensing exams.

I'm sure there will be a big push to put it online for the sake of convienence, especially with a virtual organization. If that happens you've wasted your time. A guy with a few Code Check books, some basic knowledge, and a hole in his principles could pass easily. The NHIE is taken at a neutral test site and monitored, very hard to cheat on.

Will there be more than one form of the test? If this is the only one and members can access it freely now, don't you expect many will simply work out all of the answers, wait for the test to become official, then pass with a very high grade and crow about being a "Master" inspector? I'll be honest with you Gerry, I don't think this test is nearly challenging enough to give someone such a title. I would never claim that title for myself at this point, even if I had aced it.


As to the question of Nick's NHIE quote he misunderstood the question sources that we had available, yes many questions have been reported as being from NHIE but that is only to say that we collect questions from many sources, not the least of which is from members (and my students) that have taken that exam, and every other training org does the same thing, I have very many crib sheets in my posession that were prepared by the training schools from exactly the same sources.

You never said what you thought of the practice. That "misunderstanding" aside, Mr. Gromico was quite clear in his intent to undermine the NHIE and state exams for the benefit of NACHI members. This does not strike me as appropriate or legitimate for those who seek to be "the best inspectors in the country". Shouldn't the professional organizations and institutions hold themselves to a higher standard than the private sector profiteers?

Brian G.

Link to comment
Share on other sites

Hi to all,

Brian, what is relevent or not is very subjective, I also agree with your comments about HI's having a code certification (That's my personal take).

Initially there will be one version of this exam, I do have plans to regionalize it later, and yes theroretically some one could memorize all the answers, good luck to them, I have audited every question on this exam (hundreds of questions) and I still can't ace it. As to If/how/when it will be taken and whether moderation will be a feature of it, As I said before that is undesided.

Brian, you know me better than to drag me into a Nick discussion, he is a big boy and can speak for himself, if he wishes to. All I can tell you is that this is a NACHI education project, and Nick has had no inputon it (nor did he wish to have)

Regards

Gerry

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...