Codility - Is it reasonable?

Quote:

Original post by mkuz In description Codility did not mention that they score not only correctness, but also program performance.

Why would you assume that performance is unimportant? Right off the bat I know how you coded your Fibonacci routine (recursive, no memoization), and that indicates that you might not have a great handle on algorithmic complexity. Sounds like this situation was a success for the company, as well as a learning experience for you.

Quote:

the current test is deceiving and illegal

heh :-)

You didn't get the job. Sorry to hear it. Practice up, and apply somewhere else.

I don't see how this would be illegal. You failed the tests they set you, and you didn't get the job. How is that unfair?

Honestly companies get thousands of applications a year, and many of the people applying aren't fit to work there. The bulk of the rest are about equal, with a handful being exceptional. The goal of HR departments is to cut out those who shouldn't have applied in the first place, and then find the exceptional ones. Failing any exceptional ones applying, it really becomes a game of luck for the bulk of them.

Quote:

Original post by mkuz For example, in fibonacci calculations 3 tests were marked as "time-out error".

Did you use recursion or correctly handle overlow?

Quote:

In bit count they did not mention that the value could be more than 32-bit, gave me successful test result verifying with 32-bit number and sent unsatisfactory report to my employer for tests with 1000000-bit nimber. As a result I lost this employment.

Well, 32-bit integer is a bit simple already, since much of code today is 64-bit.

How did you read the input? Did you check for overflow there? Why not? What about reading strings in C code using scanf? How do you protect from buffer overflow? Can you write an overflow-safe input reader? Would use use malloc, calloc, realloc or alloca? What are the differences between them? If all you have are 32-bit registers, how would one multiple larger numbers? What about division? How does carry flag work? How would one detect overflow? What are half- and full-adders?

Those are all related questions. One simple way is to use languages which have large numbers built in. For coding competitions, unless there is a guaranteed upper bound, I simply use java's BigInteger, if only to cover potential overflow problem.

Quote:

My opinion is that the current test is deceiving and illegal to use in hiring process.

"Illegal"? ...

Is this your first time doing the interview? At least you didn't need to take a day off, drive 2 hours, be grilled for 3 more, only to fail because you couldn't name all 67 parameters to ls.

Quote:

Test time should not be limited by 2 hours.

In normal interview you would have to solve all 3 of those assignments in 15 minutes over a phone.

Quote:

Current test is not valid for a hiring process.

Sure. And it's also not fair that bankers get $20 million in bonuses, that 200 million children will starve to death this year, that floods left 100 million homeless and that my neighbor insists on mowing lawn at 7 am on Sunday.


While it sucks not getting a job - speaking as someone from the other side, reading this post, I'd have much bigger concern about whiny attitude. No - life is not fair.

Look at it this way. Today you learned many valuable things:
- What codility is
- Hiring interviews are not necessarily fair
- Sometimes, problems require numbers that are larger than 32 bits
- It may have been good to have done some coding competitions before, even just for practice
- Position is a code monkey job - if employer can reject someone because of a trivial coding error in such conditions, they aren't interested into their 5 year project management experience (hence they have more than enough applicants and can throw them away like this)
- Unless codility publishes full test results to employer, it is a service to avoid by employers who aren't hiring in bulk
- When dealing with ASCII numbers, never assume upper bound (scanf overflow and similar)
- Life is not fair. There are people who get rejected because of their color, accent, wearing wrong brand of shoes, because the job has already been filled by CEO's brother, because interviewer has a bad day and doesn't want to spend time on you or because the job posting is a fake and merely used to justify using outsourced labor.


As a side note, google for codility+reddit. It was tested a while back, and many coders fell for exact same problem (timeout as well as overflow).

Quote:

My opinion is that the current test is deceiving and illegal... Current test is not valid for a hiring process.

Unfortunately, the company hiring you disagrees. Oh well, that's life.

It's certainly not illegal.

Once you get out of college, life is nothing but problems that you have to solve correctly despite not having all of the constraints defined.

Nothing in those tests seems unreasonable to me. They didn't explicitly spell out some of the restrictions and requirements, but that means you're supposed to ask them. In fact, most good job interviewers will explicitly leave out important requirements or details to see if you do ask. A large part of software development work is making sure you have the right requirements and understand them.

"Would a naive recursive algorithm be OK or do you want to see a O(1) memory-space solution?"
"Whats the run-time restriction?"
"How many bits are valid for the input?"

Asking these sorts of things is a plus, because it shows that you're observant, thinking about the run-time/memory-space-complexity and inputs of your algorithm.

I don't think its unreasonable, but it is a little bit underhanded not to mention that these criteria are important. An attentive candidate should probably ask about additional constraints/requirements, but interviews can be stressful and people can forget such seemingly "obvious" steps.

Were I in an interview, I would tend towards writing simpler code rather than go for something more complex (which might be faster or better handle extreme boundaries) but more likely to contain a bug.

I believe many programmers would be the same. It is ultimately the company's call. I personally think they might be losing out on good applicants who would have done well if they were aware of the judging criteria in advance - and they might be hiring applicants who routinely present over-engineered solutions to simple problems. Not everything needs to be super optimised, and "int" has a good range for 99% of use.

Without stating additional requirements, I would find it hard to fault programmers with going for the simpler solution. It is what I do myself at work.

Codility does have a free demo. You can try it out and see how it works before you do it for real. One of the more interesting aspects of it is that just by choosing certain programming languages you pass some tests automatically for "free". Ex: using python gives you automatic bignum support with integers.

Quote:

Original post by Rycross Asking these sorts of things is a plus, because it shows that you're observant, thinking about the run-time/memory-space-complexity and inputs of your algorithm.

You can't ask during automated interview.

Fibonacci is *never* coded using recursion. The only exception is writing an introduction to algorithms and data structures, where it's used to demonstrate why recursion fails miserably. This is litmus test.

Counting bits is about reliable input parsing.

Quote:

Not everything needs to be super optimised, and "int" has a good range for 99% of use.

Using C - read user's name. Same problem. How do you do it without being open to buffer overruns. Using any managed language - parse URL string. Can you ensure that the parsing algorithm will be properly bounded in time and space?

How do you count bits in a number (which comes from some input)? Read one char at a time, count bits. Unless input stated that sequence of ASCII characters will always parse into a number with range of less than 32-bits, assuming that is very flawed. Details depend on actual requirements, but I'm willing to bet that problem doesn't really require storing the number as binary or memory, or hint at that being a requirement.

Quote:

I personally think they might be losing out on good applicants who would have done well if they were aware of the judging criteria in advance

Here is what the employer sees (not codility, but similar automated tests):

1. Candidate #1      100 14min2. Candidate #2      100 19min3. Candidate #3      100 24min4. Candidate #4      100 39min5. Candidate #3      98  36min...14. You              93  56min...55. Candidate #55    2   120min56. Candidate #56    0   120min...498. Candidate #498  0   0min499. Candidate #499  0   0min500. Candidate #500  0   0min

Top 5 are selected for phone interview. #5 will be grilled about that one failed test.

Numbers are made up, but quite realistic. Half of applicants won't even take the test (they got coaxed into the test by recruiter or weren't even told to take it, or are simply spamming resumes). The remaining half will take it, try the first assignment, then say screw that.

#14 might be a good developer. But since company will need to dedicate a full day to personally interviewing the 5 applicants, it makes sense to choose best-of-breed to start off.

Finally - if person's experience matters even in slightest, their first encounter will not be a dumb coding test, but they will be brought in because of past history, connections, references or related work. But for entry-level coding jobs - code is all that matters.


Sure, the person might learn quickly about Fib, they might learn about input parsing, or they might not. But the higher their initial level, the less burden they will be on others, and the more they can focus on business parts which actually make money. Especially since it's possible they won't be doing coding at all, but will instead be trained in more important business domain related skills, yet understand the details of actual programming.

I don't believe in "losing good applicants" anymore. If anyone feels they can't handle such tests, then they'll need to learn how to bypass HR, and get into company directly. That is how non-entry-level jobs are handled anyway. As long as you are asking someone to employ you, you have no bargaining position, and no matter how good, there are more than enough people better than you.

Now we can either throw around words like "illegal", or try to understand how the system works to deal with it effectively. Fortunately, hiring is messed up enough that simply trying 500 times will net a job.

But here is another perspective - assuming you passed that test with 100 points. How would you feel about your coworkers who only scored 74, 58 and 12, but were hired anyway?

I admit that Codility is a tad harsh, but overall, it is a great tool. You should have good programming habits all the time, not just when asked. Expect the worst case scenario unless instructed otherwise. You being upset over lack of instruction over how big the numbers become is like a prospective musician whining, "Well, you didn't tell me to play my best! You just told me to play a scale, so I did!"

Quote:

Original post by Antheus You can't ask during automated interview.

I missed that it was automated.

Quote:

Original post by Antheus Fibonacci is *never* coded using recursion. The only exception is writing an introduction to algorithms and data structures, where it's used to demonstrate why recursion fails miserably. This is litmus test.

In interviews, if we ask people about fibonacci we will usually accept recursion as a first "naive" attempt. We will then follow up asking about its limitations and perhaps ask them to code it in constant memory space.

Obviously, you would never use a recursive solution in an actual piece of software, but in the context of the interview we usually don't mind an interviewee putting out a naive or trivial solution as long as he recognizes its not optimal, why its not, and can explain how to do it optimally.

Quote:

Original post by TheBuzzSaw "Well, you didn't tell me to play my best! You just told me to play a scale, so I did!"

Two explorers are sleeping in a tent. They hear a lion outside. First one tries to run away but sees the other one putting on sneakers.
- Why are you putting on sneakers? You can't outrun a lion even with them
- I don't need to outrun a lion - just you

Being best is about being just slightly better than the competition.

Testing candidates using really simple problems is a good idea in general. One problem that I see with this approach though is that this only tests coding ability but not debugging ability - even though debugging ability is probably much, much more important in the long run.

It's possible that a company that relies only on this test might miss out on an awesome debugger, just because she forgot to check for a stupid boundary case in one of the tests (it is my understanding that once you submitted your solution, you cannot go back when you see that a test fails, even though you might immediately realize what your mistake was).

Quote:

Original post by mkuz In description Codility did not mention that they score not only correctness, but also program performance.

Of course. Employers are looking for staff who will do their best work all the time, not just when they are explicitly asked. You're interviewing/testing for a job; it should be obvious that you need to do the most efficient solution, not just he minimum work necessary to solve the problem.

I tried their sample tests and used this code:



It only got 75 out of 100 how would you score higher?

Here is a thread on codility where people discuss the sample problem.

I'm candidate #1 xD lololol

Never knew about Codility. Pretty cool stuff. Prolly could have written it in less time, but I wanted to make sure it had every optimization I could imagine.

Machine-scored coding tests might prevent the company from wasting time on bad candidates, but it'll also (at least) occasionally prevent the company from getting the best of the best. Judging efficiency without listing it as a grading point sounds like a bad idea to me - you'll end up paying more per unit work due to premature optimization, associated debugging, etc.

IME, it is far better to use open-ended pre-screen questions that are judged by humans. For example, "Describe what happens when you open a URL in your browser." This works very, very well for my employer.