I have a webcrawling software, written in Java, that goes through any given website, and collects data. So far, so good. I use the JSoup API for ease, but that’s irrelevant. My problem lies in encoding.
When I run the Jar file on the server (using java -jar Crawler.jar), it captures data perfectly – including international accentuation, like á, é, and so on, so forth. HOWEVER… When I schedule it through crontab, it just screws up every non-basic character! What could be the problem here? I’m really at a loss, an answer would truly save my skin.
EDIT – A friend suggested I checked out the locale running on Cron, using this code:
*/1 * * * * locale > /home/user/locale.ouput
I did, and it seems that crontab is using a POSIX locale, as opposed to the system settings, that are UTF-8. I’ll look on how to change it to UTF-8. Any tips would be appreciated!
Set the character encoding explicitly whenever you convert bytes to or from characters. For example, in the String constructor, String.getBytes(), InputStreamReader constructor, etc.
If that isn’t enough, try setting the LANG environment variable to a different value inside crontab before it launches Java. For example, you could use