Fast, Efficient Data Storage on an Arduino


Due to WordPress’s abysmal handling of code blocks this blog post is now hosted at https://majenko.co.uk/blog/

Logging data on an Arduino is very much a trivial every-day task. Connect an SD card, open a file, and start printing data to it.

For many people that is good enough. It results in nice easily readable (by us humans) data.

But it’s not fast. It’s not efficient. It’s perfectly fine for things like logging temperature every hour, or barometric pressure every 5 minutes, etc. But when you have large amounts of data to store very rapidly you have to think a little differently.

I came across a situation recently where it was necessary to store lots of data values very rapidly and be able to read them back again at a similar speed.  Now that could be done by writing those values to the file as text, maybe as a Comma Separated Values file (CSV), which is simple enough, but when it comes to reading that data back in on an Arduino things get decidedly complex. And complex means lots of code and large processing overheads.

So what is needed is a way of storing the data in such a way that it is trivial to read it back in.  So for this I am going to give you two small phrases that sum everything up quite neatly:

  • Human Readable
  • Machine Readable

That describes two types of file.  Human Readable files are things like text files, CSV files, etc. You can open them in a simple text editor and you understand what they are. It’s just text.  However a computer has a hard time understanding them.  The opposite end of the spectrum is Machine Readable files. These can’t be understood by a (normal) human being. You open them in a text editor and all you see is gibberish. It takes a special computer program to interpret them and display a representation of that data for a human to make head or tail of it.  A good example is a graphics file – say a PNG file.  Here is a PNG file opened in GEdit on Linux:gedit

As you can see it’s just nonsense. However open it with a graphics program and that program reads the file and uses it to create a picture you can see.

Such data is said to be Binary. It is important to note, of course, that binary contains text. That is text files are just a subset of binary files.  In a binary file each entry (byte) can contain a value between 0 and 255. Text files just map the letters we all know and love to numbers within that range of 0-255. So there is absolutely no difference between a text and binary file – its is just that a binary file can contain more data outside the range of human understanding. For instance in the PNG file above you can see the word “PNG”, and “HIDAT” and other letters and numbers besides the stuff that you can’t understand. So a binary file that contains only bytes in the human readable range (also known as ASCII values – the American Standard Code for Information Interchange) we choose to call a text file.

So how does this help us store data more efficiently on an Arduino? Well, simply by stepping away from the limiting factors of the human interpretation of data and using a purely machine readable file.

For this we are going to use a struct. In C a struct is a method of grouping different variables together into one single unit. You can think of it as a bit like an entry in a database, where you might have a name, address, town, postal code, all as different fields within it.  In C those fields are variables, and the struct is the record.  Let’s take an example:

struct datastore {
    uint16_t adc1;
    uint16_t adc2;
    float voltage;
    float current;
};


There we have defined a structure that contains four different values within it. Each value has its own associated data type, just like normal variables.  The whole structure itself is, as well, a new data type. You can make new variables from it, like:

struct datastore myData;


You now have a new variable called myData which itself has 4 sub-variables. You access those using a “.” and the name:

myData.adc1 = analogRead(0);
myData.adc2 = analogRead(1);
myData.voltage = myData.adc1/1024.0*5.0;
myData.current = myData.adc2/10000.0*23.4429;


It’s a useful technique in its own right for grouping different variables together that are related, but it’s real power comes when you get under the hood and look at what is actually happening.  Not only are the sub-variables grouped together within an umbrella name like that, but they are also grouped together in memory. And in a very specific way as well. The order they are specified in the structure is the order they are held in memory. For instance the struct above might look like this in memory:

StructureMemory

Each square is one byte in memory. As you can see the uint16_t values (same as an unsigned int on the Arduino – I’ll cover why using uint16_t and not unsigned int is important a little later) use two bytes, and the float values use 4 bytes each. That gives a total of 12 bytes. And of course it is perfectly possible to access those raw bytes of data should you wish to.

And we wish to – although not directly.

There is another very useful function in C called “sizeof(var)”. That tells you how big variables are. For instance it would return 2 for a uint16_t, 4 for a float, etc. For our struct it would return 12.

So now what if we were to write those 12 raw bytes direct to the SD card instead of a textual representation of the numbers?  We would end up with a file that was 12 bytes long. Write it twice and we would have a file that was 24 bytes long. Three times and it would be 36 bytes long.

The SD library on the Arduino supports that kind of writing perfectly well. You don’t have to do anything special when creating or opening the file. All the magic happens when you tell it to just write a bunch of bytes instead of text:

myFile.write((const uint8_t *)&myData, sizeof(myData));

Yes, I know, that looks a little cryptic, so I’ll break it down for you so you can see what is going on here.

“&myData” gets the address in memory where the data is stored.  It is intrinsically a “struct datastore *” type. The write function doesn’t like that type, so we need to change it. That is called casting, and we want to cast it to an unsigned byte pointer type, so prepend it with:

(const uint8_t *) &myData

The write function now sees it as an array of bytes. Clever, eh? Along side that we need to tell the write function how many bytes to write, and for that we can use the handy sizeof() function I mentioned before.

So let’s roll that all into a complete example:

#include 
#include 

const int chipSelect = 4;
File dataFile;

struct datastore {
    uint16_t adc1;
    uint16_t adc2;
    float voltage;
    float current;
};

void setup() {
    Serial.begin(9600);
    Serial.print("Initializing SD card...");
    pinMode(10, OUTPUT);

    if (!SD.begin(chipSelect)) {
        Serial.println("Card failed, or not present");
        return;
    }

    Serial.println("card initialized.");
    dataFile = SD.open("datalog.dat", FILE_WRITE);
}

void loop() {
    struct datastore myData;
    myData.adc1 = analogRead(0);
    myData.adc2 = analogRead(1);
    myData.voltage = myData.adc1 / 1023.0 * 5.0;
    myData.current = myData.adc2 / 10000.0 * 23.4429;
    dataFile.write((const uint8_t *)&myData, sizeof(myData));
    delay(50);
}


So now we are filling our SD card with raw binary data. But what can we do with it? We can’t look at it, it will just be meaningless to us.  So we need the Arduino to read it for us. And that is just as simple. There is a “read” equivalent to the “write” function we used above where we can tell it to read bytes into an array – and that array can be our struct cast as before:

myFile.read((uint8_t *)&myData, sizeof(myData));

That will read the 12 bytes from the SD card and reconstruct your structure for you, all magically and without you needing to do any interpreting of numbers or symbols. So we can take the example we already have and change it into a reading example very very simply:

#include 
#include 

const int chipSelect = 4;
File dataFile;

struct datastore {
    uint16_t adc1;
    uint16_t adc2;
    float voltage;
    float current;
};

void setup() {
    Serial.begin(9600);
    Serial.print("Initializing SD card...");
    pinMode(10, OUTPUT);

    if (!SD.begin(chipSelect)) {
        Serial.println("Card failed, or not present");
        return;
    }

    Serial.println("card initialized.");
    dataFile = SD.open("datalog.dat", FILE_READ);
}

void loop() {
    if (dataFile.available()) {
        struct datastore myData;
        dataFile.read((uint8_t *)&myData, sizeof(myData));
        analogWrite(5, map(myData.adc1, 0, 1023, 0, 255));
        analogWrite(6, map(myData.adc2, 0, 1023, 0, 255));
        Serial.print(myData.voltage, 4);
        Serial.print(" ");
        Serial.println(myData.current, 4);
        delay(50);
    }
}


So simple. No need to try and understand the data, the Arduino already knows what it is.

There are a couple of gotchas though with this method.

  1. The data structure mustn’t change, or you won’t be able to read old data. It relies on the struct always being the same size and with the same variables in it. If you suspect that you may want to add more variables to the structure at a later date you should reserve room for them in the structure right from the start.
  2. Different chips, boards and computers treat different variables in different ways. For instance on an Arduino Uno an int is 2 bytes, but on a Due it’s 4 bytes. That is why it is important to use things like uint16_t instead of “unsigned int” – it tells the system precisely how big a variable to use and all systems will then use the same size.
  3. Carrying on from 2 is the problem of endianness. Not only do different systems have different sizes for different variables, but there are a number of ways of arranging them in memory – for instance the uint16_t has two bytes, but which byte is which? There are two common endiannesses – big-endian and little-endian, and making sure that you convert between them on different systems is vital or your data will just come out as nonsense. For instance, take the Arduino Yun. The ATMega32U4 chip is little-endian. That means that in a 2-byte variable like the uint16_t it stores the least significant byte first. The Linux portion, though, happens to be big-endian. That means it stores the most significant byte first. So to read the data written by the ATMega32U4 on the Linux side you will have to manually swap the bytes around when reading from the structure.

56 thoughts on “Fast, Efficient Data Storage on an Arduino

  1. apicus

    you probably mean :

    analogWrite(5, map(myData.adc1, 0, 1023, 0, 255));
    analogWrite(6, map(myData.adc2, 0, 1023, 0, 255));

    Like

    Reply
  2. Bertie

    Hi, would the approach be similar for saving a class instead of a struct?

    I have a main class that has attributes which are ints, floats, structs, strings and pointers to other classes.

    This is all new to me (Arduino, C++) so excuse any naivety. I’m also still waiting on my SD module so I haven’t had the chance to play around with this yet. Anyway my worries/ thoughts/questions is:

    Is the size of each instance of my class fixed? My gut tells me “no”, and that this precludes mimicking the example above.

    If I am right then I am thinking that I need to get the actual size of each instance of myClass and save that to the SD as well so that I can read back the same data that was saved…. Am I on the right lines?

    Great write-up BTW. Better than most of the explanations I have read so far today. Thanks.

    Like

    Reply
  3. Pingback: A DIY Arduino data logger: Build Instructions – Part 4 (Power Optimization) | Arduino based underwater sensors

  4. Reda

    hi
    thanks for your post ,
    i.m trying to build an electronic access door system with Arduino mega , so i decided to store users data in a binary file on SD Card which attached to Adafruit TFT touch screen
    my problem is :
    when i open the file to read and search for a user it works and retrieve the related data but when i try to open the file again to make another search for another user i got this message :
    “Card failed, or not present”
    and even screen stops to respond,but every thing works again after restarting the MEGA board

    Like

    Reply
    1. majenko Post author

      Unfortunately comments here aren’t the best place for debugging code. I would suggest you ask a question on arduino.stackexchange.com were you can post your code so we can see what is wrong with it. One thing I would suggest though is to not open/close the file, but to only open it once at the beginning of your program (in setup()) and then use seek() to jump to different places within it. For instance “rewinding” the file to the beginning can be done with “myFile.seek(0);”. You can also jump to a specific record number within the file with “myFile.seek(sizeof(struct myDataStruct) * recordNumber);”

      Like

      Reply
  5. El

    You are my personal hero.
    I was thinking about that for days, and you not only giving a solution … I understand it 🙂
    I was dealing with arrays and huge for loops on my Arduino, and now my save/load functions are clean, easy to read, powerful and have 20 lines 😉
    Thank you very much!

    Like

    Reply
  6. tingeman

    Thanks, this was very helpful! Upon trying this with a struct containing one uint32_t and three uint16_t members, each record written end up having two extra bytes of value 0x00. I assume this has to do with padding of the struct. Do you observe this as well? Can I expect the padding to always be at the end of the struct, so that I could simply write always 10 bytes (instead of using “sizeof”)?

    Like

    Reply
    1. majenko Post author

      On an 8-bit system I wouldn’t expect to see any padding. However you can “pack” the struct using `struct blah { … } __attribute__((packed));` which will eradicate any padding bytes and also save you some space.

      Like

      Reply
      1. tingeman

        Thanks, I’m using a 32 bit NavSpark-GL arduino compatible board, so that probably explains the packing. The “__attribute__((packed))” works perfect!
        I’m not concerned about memory or SD storage capacity – more about speed, as I am trying to obtain high-frequency logging from an external sensor. Do you know of any downsides to packing the struct? (there must be a good reason it is not packed by default).
        Thanks again for your help!

        Like

      2. majenko Post author

        Packing can introduce a small amount of extra processing to access the structure content as it has to extract parts of words and recombine them again. Some architectures provide byte and half-word access instructions which speeds things up somewhat. Not sure what the NavSpark has though.

        Like

  7. oldmanegan

    You have a small typo… see last line, rest here for context and help finding…
    “So now we are filling our SD card with raw binary data. But what can we do with it? We can’t look at it, it will just be meaningless to us. So we need the Arduino to read it for us. And that is just as simple. There is a “read” equivalent to the “write” function we used above where we can tell it to read bytes into an array – and that array can be our struct cast as before:

    myFile.read((uint8_t *)&myData, sizeof(mydata));”. Should be myData, not my data…

    Great work and many thanks!

    Like

    Reply
  8. Pingback: Arduino Data Logger: 2017 Build Update | Arduino based underwater sensors

  9. Karl-Johan Schmidt

    I tried using your example, but I get an error at dataFile = SD.open(“datalog.datLE_READ)
    Isn’t there missing a “, or am I wrong?
    If i add the “, it executes, but I’m not able to read any data

    Like

    Reply
      1. Watson

        This is not a copy-paste error but a browser error.
        In Microsoft Edge you will see:

        dataFile = SD.open(“datalog.datLE_WRITE);

        Like

      1. majenko Post author

        The only sign of “datLE” on this page is in the comments. I have checked, double checked, examined the HTML code behind it all, and there is nothing wrong with the formatting of the example. If you like I can take a photograph of it to prove it.

        Like

  10. Mathias Wilhelm

    Salut,
    I checked with Chrome, Firefox and IE and in all three I get the text SD.open(“datalog.dat”, FILE_READ); shortened to SD.open(“datalog.datLE_READ);
    The HTML code however is perfectly ok showingthe original string. And I cannot spot the reason for the browsers to change the displayed text!
    I checked the OS: same effect on Windows7, Windows10 and Linux (Ubuntu Mate with Firefox)
    This is really strange as the source code in HTML is not in any kind special. It happens with the Mesage “Initializing SD Card” as well – it gets reduced to “Initializingard”

    Like

    Reply
    1. majenko Post author

      That is decidedly odd. I am Chrome on Ubuntu (with Awesome not Mate) and it’s perfectly fine. I’ll try some tag tweaks, see if that helps things.

      Like

      Reply
  11. Mathias Wilhelm

    Salut Matt,
    that changed it to display everything correctly – but I still do not understand why without the code flag should have this impact. The souce code does not show any scripts that woud do that … Must be Sauron in some way as the first statement sounded a bit like Isengard 😉
    Ciao, Mathias

    Like

    Reply
  12. Jann Antons

    Hey there. Thanks for your work and Code provided.
    I love the idea of struct. I try to use it for my Weather Station. I like to store the overall max and min values “ever” measured. So i calculate the max value and store it. Next step is to see if the “new” max value is larger then the stored value. AND there i have a problem.

    float a = (maxtemp);
    float b = (myData.maxtemp);

    Serial.println(“a and b Values:”);
    Serial.println(a);
    Serial.println(b);
    Serial.println(“a nd b END”);

    if (a = b) Serial.println(“a=b”);
    if (a < b) Serial.println("a b) Serial.println(“a>b”);

    Even if a is unequal b the Serial Monitor prints a=b

    I like to store the new max value to SD and take that as a new max for the next loop measurement. So if i have a power lost situation i still have that max ever value. The day by day max will be erased at midnight.

    //_______________________________________________START
    //Full code:
    #include // SD Karten Bibliothek
    #include // SPI für SD Kartenleser
    #include “DHT.h” //Bibliothek für DHT einbinden
    #define DHTPIN 2 // PIN an dem der DHT22 Ausgang angeschlossen ist DIGITAL 2!!!!
    #define DHTTYPE DHT22 // DHT 22 Als Sensor definieren
    DHT dht(DHTPIN, DHTTYPE);

    const int chipSelect = 4;
    File dataFile;

    float temperature,maxtemp,mintemp = 0;
    float h,maxhum,minhum = 0;

    // Here starts the Magic of struct
    struct datastore {
    //float temperature;
    float maxtemp;
    float h;
    };

    void setup()
    {
    dht.begin();

    Serial.begin(9600);
    Serial.print(“Initializing SD card…”);
    pinMode(10, OUTPUT);

    if (!SD.begin(chipSelect)) {
    Serial.println(“Card failed, or not present”);
    return;
    }

    Serial.println(“card initialized.”);
    dataFile = SD.open(“datalog.dat”, FILE_WRITE);

    struct datastore myData;
    dataFile.read((uint8_t *)&myData, sizeof(myData));
    Serial.println(“Stored Value: “);
    Serial.println(myData.maxtemp, 2);
    Serial.println(myData.h,2);

    maxtemp=dht.readTemperature();
    mintemp=dht.readTemperature();
    maxhum = dht.readHumidity();
    minhum = dht.readHumidity();

    }

    void loop() {

    dataFile = SD.open(“datalog.dat”, FILE_WRITE);

    float temperature = dht.readTemperature(); // Temperatur in Celsius
    float h = dht.readHumidity(); //Read Humidity

    // min and max values calculation
    if ( temperature > maxtemp ) maxtemp = temperature;
    if ( temperature maxhum ) maxhum = h;
    if ( h < minhum ) minhum = h;

    Serial.println("Loop Starts here");

    Serial.print("Temp: ");
    Serial.print(temperature);
    Serial.print(" *C ");
    Serial.print("Hum: ");
    Serial.print(h);
    Serial.print(" %\t");

    Serial.print("TempMax:");
    Serial.print(maxtemp);
    Serial.print(" *C ");

    // Write Data
    struct datastore myData;

    myData.maxtemp = maxtemp; //temperature = temperature;
    myData.h = h;
    dataFile.write((const uint8_t *)&myData, sizeof(myData));
    delay(500);
    Serial.println("Data has been written to SD Card");

    dataFile.close(); // dataFile close for next step – open and read

    delay(3000);

    Serial.println("Open SD Card for reading…"); //Only debug help
    dataFile = SD.open("datalog.dat", FILE_READ); //Open datalog to READ data

    delay(3000); // only test

    //Read values START
    if (dataFile.available()) {
    Serial.println("datafile available"); // Only Info…
    struct datastore myData; // Daten read…
    dataFile.read((uint8_t *)&myData, sizeof(myData)); // Format read
    Serial.println("Stored Values: ");
    Serial.println(myData.maxtemp, 2);
    Serial.println(myData.h, 2);
    delay(500);

    // Read Values END
    }

    delay(2000);
    float a = (maxtemp);
    float b = (myData.maxtemp);

    Serial.println("a and b Values:");
    Serial.println(a);
    Serial.println(b);
    Serial.println("a nd b END");

    if (a = b) Serial.println("a=b");
    if (a < b) Serial.println("a
    b) Serial.println(“a>b”);

    /*
    //if ( maxtemp = (myData.temperature) )
    {
    // Store new max value here…
    Serial.println(“maxtemp = stored data”);

    }
    Serial.println(“maxtempstored Data”);

    Serial.println(“Ende Loop”);
    */
    delay(5000);
    }
    //___________________________________________END

    Any ideas how to fix it? I would really appreciate any help.
    Greetings from North Germany.

    Jann

    Like

    Reply
      1. Jann Antons

        I am a little bit ashamed of myself. Thank you very much. That did the trick. Unbelievable i did not see this. I really appreciate your help. Thanks again from North Germany.

        Thanks, Jann

        Like

  13. allearsaudio

    Much thanks, excellent explanation and example code. I’ve been struggling with this and searching for answers for a few days now, and this is by far the clearest explanation I’ve seen. Just what I needed!

    Like

    Reply
  14. Richard Yates

    This is great, just what i needed, i had been struggling with how to correctly write binary data that comes in from a device connected to the serial port. What is the best way to deal with the input side (binary data from serial port)? Is it better to read the incoming binary data into an array and then write the array to a file on SD card when array is full or write binary data directly from serial to SD card?

    Thanks.

    Like

    Reply
  15. Hamid

    i firstly i should tell that your text is very helpful then i’d like to ask a question about the saving of the struct.
    your code overwrite the next analog read (next struct) over the last struct what if i have to save the structs one after the other. how can address the next struct to be written. can you make an example?

    Like

    Reply
  16. PS

    Great article, thanks. What if somebody is storing 1000 structs, then they want to read back struct number 537 without wasting time reading structs 1 through 536? Would it be something like this?: dataFile.read((uint8_t *)&(536*12), 12); With a struct of 12 bytes like in the example.

    Like

    Reply
    1. majenko Post author

      No, you’d use seek to jump to a multiple of the struct size. myFile.seek(536*size of(struct myStruct)); Then you can just read the single struct of data.

      Like

      Reply
  17. Ritz

    I want to store 512 data together at once in sd card using arduino.if it is one by one it is slow.so how can I store baud data in sd card?What is the required code?

    Like

    Reply
  18. Ritz

    I want to store 512 data together at once in sd card using arduino.if it is one by one it is slow.so how can I store baud data in sd card?What is the required code??

    Like

    Reply
  19. JamesONeil

    Hi ! Thanks for sharing this nice trick ! I am currently trying to acquire data from a 3-axis accelerometer at least each ms but with the RTC module running in parallel and later in project other captors I might encounter some obstacles… I would like to try to save the data as binary like you said in a file at first and then read this file and create a new one on the sd that is exploitable by human in a csv format. Do you think it is a good idea to process in two times this way so I can get the maximum writing speed from my sd ? Because I don’t see how to apply what you did for my case. I need to write fastly data but also I need it to be in a file that can be read by a human thus the binary data are a problem for a one time shot. If I write directly the data in a csv format the acquisition time is not fast enought for my data logger…

    Thank you for your help !

    Like

    Reply
    1. majenko Post author

      Unless you have “dead” times when you’re not acquiring, where you could process the data into a readable format, then no – it will only make things even slower. You would typically write a piece of off-line conversion software on your PC to take the binary file and spit out a human readable format.

      Like

      Reply
      1. JamesONeil

        Ok thanks for your fast answer ! I won’t have “dead” times to process the data during acquisition…So according to you I should register directly the data in a csv format ? What do you advise me to do to write the data as fast as possible ? A buffer ? I was trying this method on a single data acquisition via an analog port and it increased significantly my acquisition time. However in the real process I have many sensors via I²C and I have difficulties to treat all the data in one buffer.
        Thanks again for your help !

        Like

  20. majenko Post author

    You should store it in binary format as suggested, then when you want to read the data you pass it through a program on your computer that reads the binary format and outputs a text format. You can write it in whatever language you like that you feel most comfortable in (e.g., python). My personal choice would be to use C or C++ since you can then use the same struct for reading as you use for writing, and other languages would require you to work with individual bytes piecing things back together.

    Like

    Reply
  21. JamesONeil

    So I tried like you said by following your example. First I wrote the data with arduino in a binary format with a structure made of 3 floats (one for each axis of acceleration) so the size of my struc is 12. Then I wrote the read part of the program in an other folder of arduino but when I display the value that is being read I got really weird values of 0.00 and ovf and so on… Obviously there is something I am doing wrong in the way I read the data so I put here the piece of code of the loop to read :
    void loop() {
    if (myFile.available()) {
    struct data myData_read; // create a struct containing the 3 floats
    myFile.read((uint8_t *)&myData_read, sizeof(myData_read)); // read the pack of 12 bytes
    Serial.print(myData_read.acc_x,4);
    Serial.print(“;”);
    Serial.print(myData_read.acc_y,4);
    Serial.print(“;”);
    Serial.print(myData_read.acc_z,4);
    Serial.println(“;”);
    delay(50);
    Serial.println(“END”);
    }
    }

    And here is how I write the data in the file :
    void take_measure(){
    struct data myData;
    myData.acc_x = get_accel_MMA8451(X);
    myData.acc_y = get_accel_MMA8451(Y);
    myData.acc_z = get_accel_MMA8451(Z);
    myFile.write((const uint8_t *)&myData, sizeof(myData));
    Serial.println(myData.acc_x, 2); // these 3 data are correct by checking them on Serial monitor
    Serial.println(myData.acc_y, 2);
    Serial.println(myData.acc_z, 2);
    }

    Thanks a lot for your help it would be great if I am able to convert correctly my data !

    Like

    Reply
  22. verrocchio daniele

    Dea,
    thank you for your very helpful example; unfortunatly, dosn’t work for my, I trying on Adafruit M0 express and M4 express. the file .dat i created right but it is empty, 0byte and no daa inside. I copy exctly your code without change.
    thank you
    Daniele

    Like

    Reply
    1. majenko Post author

      It could then be that you aren’t closing the file before removing the SD card. If you write less than the size of the internal buffer in the M0’s SD library (512 bytes on the 8 bit Arduinos – no idea how big on the M0) then nothing gets written until you either flush or close the file.

      Like

      Reply
  23. Leonardo Bravo

    Hi,
    Nice post, and thank you for sharing this information.
    I was about to write the code on my arduino; however, I realized that you never close the file, so, is it necessary to close it?
    Another question, I am working with 15 sensors, is it recommendable to use a struct with that number of sensor?
    Thank you for your help.

    Like

    Reply
    1. majenko Post author

      Sure, a struct for that is ideal. I don’t close the file simply because this is just an example. You should close the file (or flush the file which is like closing but without actually closing the file, so you don’t need to open it again) when you aren’t writing data fast to the file. If you have a moment where there is some “slack” time then slip a `.flush()` in there to keep the data on the card in sync.

      Like

      Reply
    1. majenko Post author

      You close the file at a time that is appropriate for your program. It should be closed (or at least flushed) before the card is removed. Ideally at a time when you’re not blasting data at it.

      Like

      Reply
  24. عبدالرحمن الزهراني Abdulrahman Alzhrani

    Thanks majenko, you’re a lot of inspiration to me and you also helped me many times in Arduino exchange. I just found your website and it’s really interesting !

    Like

    Reply
  25. Sewan Fan

    Hi MT,

    Thanks for the struct example.

    In the loop {}, struct is defined or declared as

    struct datastore myData;

    Would this be the similar to something like

    struct datastore *myData_pointer;
    myData_pointer = malloc( sizeof( struct datastore ) );

    Above code sample found in “On to C” page 119 modified for this post.

    Placed in loop{}, would struct datastore myData keep growing and lessen the available memory for use?

    We are planning to store data quickly using a TinyDuino and SD card combination for air rocket launches with about 5 seconds of flight.

    Thanks for your reply.

    S. Fan

    Like

    Reply
    1. majenko Post author

      Yes, it’s similar, but the storage location in memory differs. It’s not recommended to use dynamic memory allocation on small microcontrollers as the lack of heap space means heap fragmentation is a real problem and can lead to random crashing.

      Like

      Reply
  26. Gejakem

    Hello Majenko,

    Good post. Thanks. Reading through Arduino-examples you usually are confronted with writing human readable stuff to Arduino. Your post gives a very useful and different insight on another method.
    The title of your post says “Fast, Efficient Data Storage on an Arduino”.
    I get the Efficient-part. I think the method of writing structs takes less space on the card.
    I’m wondering about the Fast-part.
    Is it just the additional speed you gain from writing/reading less bytes of may there be an additional speed-bonus because of writing the structs?

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s