The problem has more than one solution as far as I can see:
the sum is a square, so any square number is a possible solution for the amount of money. With the condition that the elder brother took one more 10$ Bill we can conclude that any square number can be excluded which statisfies the condition int(number/10) even.
That leaves the solutions: 16 (one bill for the elder,none for the younger, check over 3$) I shorten this to (1,0,3)
since there is an infinite number of squares fulfilling the condition the number of solutions is infinite.
However there is probably a value of a sheep in a certain range, picking the right square. Or there is a minimum amount of money a check can be written out on....