Question

I’m really new to C(I have been learning Cuda and wanted to learn C so I can run everything together instead of generating data in Java/Python and copy/pasting it manually to my Cuda program to play with). I am trying to open a large file(20+gb) and parse some data but because I was having problems I decided to try to break down my problem first to verify I can open and read line by line a file and then in another file take a sample of the output of a line and try to parse it, then if all goes well I can simply bring them together. I managed(after a bit of a struggle) to get each part working but when I combine them together it doesn’t work(I tried in both eclipse & QT creator. QT creator says it exited unexpectedly and eclipse just stops..).

Here’s the parsing part(it works exactly as I want):

#include <string.h>
#include <stdio.h>

int main()
{
    char line[1024] = "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 4],";
    size_t len = strlen(line);
    memmove(line, line+1, len-3);
    line[len-3] = 0;
    printf("%s \n",line);


    char str[100], *s = str, *t = NULL;

    strcpy(str, line);
    while ((t = strtok(s, " ,")) != NULL) {
        s = NULL;
        printf(":%s:\n", t);
    }
    return 0;
}

The data in variable line is exactly as I get if I print each line of a file. But when I copy/paste it into my main program then it doesn’t work(QT shows nothing and eclipse just shows the first line). This code is just a few file IO commands that are wrapped around the commands above(everything within the while loop is the exact same as above)

#include <stdio.h>
#include <stdlib.h>

int main() {

    char line[1024];
    FILE *fp = fopen("/Users/me/Desktop/output.txt","r");

    printf("Starting.. \n");

    int count = 0;
    int list[30]; //items will be stored here

    while(fgets(line, 1024, fp) != NULL){
        count++;
        //char line[1024] = "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 4],";
        size_t len = strlen(line);
        memmove(line, line+1, len-4);
        line[len-4] = 0;
        printf("%s \n",line);

        char *s = str, *t = NULL;

        strcpy(str, line);
        while ((t = strtok(s, " ,")) != NULL) {
            s = NULL;
            printf(":%s:\n", t);
        }
    }
    printf(" num of lines is %i \n", count);
    return 0;
}

eclipse output:

Starting.. 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 4 

I can’t figure out what I’m doing wrong. I’m using a mac with gcc 4.2, if that helps. The only thing I can think of is maybe I’m doing something wrong the way I parse the list variable so its not accessible later on(I’m not sure its a wild guess). Is there anything I’m missing?

EDIT: if I comment out this block of code, it goes through the file(but doesn’t parse):

while ((t = strtok(s, " ,")) != NULL) {
    s = NULL;
    printf(":%s:\n", t);
}

the contents of the file is basically:

[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 4],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 2, 2, 4],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 2, 2, 2, 2, 4],

and so on(20+gb of that).

EDIT2: I just wanted to test if it was with my mac or not and I tried it on centos 5(installed it just to test) and I got the same error when running from command line except at the end it said Segmentation fault. All I did was copy the sample output above into a file, and changed the path in my code above and ran it.

Answer 1

I got it! Try inserting #include <string.h> with the other includes. I don’t know why you were using functions <string.h> it without including it but it caused incorrect linkage. The compiler issued the following warnings:

test2.c:17: warning: incompatible implicit declaration of built-in function ‘strlen’
test2.c:18: warning: incompatible implicit declaration of built-in function ‘memmove’
test2.c:24: warning: incompatible implicit declaration of built-in function ‘strcpy’
test2.c:25: warning: assignment makes pointer from integer without a cast

The first three warnings indicate that you did not include the header file.
The last one just goes to show that without declarations, functions are assumed to take and return integers (this is for backwards compatibility with ancient C code).

Don’t forget that fgets includes the newline character at the end of the line. You need to account for that by changing len-3 to len-4.

Also, don’t forget to fclose the file when you are done with it.

Also, your memmove is unnecessary as you can just point to the second character:

char *moved_line = line + 1;