I fully agree that pencils are fine.
Having said that, they are not anywhere close to being on my list of toys.
With great luck, none of our Amazons have ever shown any interest in them. If they did, I would provide them with the metal and rubber removed based on potential problems with the metal: sharp (cutting issues), plating (heavy metal issues), etc... Second, look for pencils that are not painted. The problem with painted pencils is unless they are produced 'in' North America or Western Europe there is little assurance as to what the paint contains.
If memory serves, Artists pencils are not painted.
Consider small diameter dowels cut to length.